types of words identified as unknown by l2 learners when reading

13
Types of words identified as unknown by L2 learners when reading Dale Brown * Graduate School of Language and Culture, Osaka University, Machikaneyama-cho 18, Toyonaka-shi 560-0043, Japan Received 20 February 2013; revised 23 July 2013; accepted 7 October 2013 Available online 25 October 2013 Abstract In determining which words are likely to cause problems for learners in reading, the computer-based lexical profiling of texts has become routine. This study investigates the nature of items marked as unknown by two groups of learners (n ¼ 46) when reading, with reference to the assumptions behind lexical profiling. The first assumption, that less frequent items are likely to be unknown, is supported by the results in that significantly more low frequency words were marked as unknown. The second assumption, regarding the use of the word family as the unit of counting for lexical profiling, is shown to be problematic. A significantly greater proportion of the higher frequency words marked were found to be inflected or derived forms. The third assumption, that few problems stem from the fact that computers can only recognise strings of characters, may be warranted. Relatively few of the higher frequency words that were marked occurred in the reading texts in ways likely to be unfamiliar to the participants. The study thus concludes that in using computer-based profiling of texts to judge which words cause problems for learners, the primary issue is the use of the word family as the unit of counting. Ó 2013 Elsevier Ltd. All rights reserved. Keywords: Vocabulary; Reading; Text coverage; Lexical profiling; Word lists; Word families; Frequency effects; Word frequency 1. Introduction In considering the types of words which cause problems for learners when reading in a second language, perhaps the commonsense response is that words which are less frequent in the language are the primary problem. This line of thinking can be easily operationalised today thanks to the development of lexical profiling. Lexical profiling involves examining the frequency level of each word in a text, allowing the less frequent items to be quickly identified. Lexical profiling can be conducted using freely available, simple tools such as the computer programs Range (Nation & Heatley, 2002) and AntWordProfiler (Anthony, 2013), with Cobb (2010) reporting that his online version of Range (www.lextutor.ca/vp/) has attracted pedagogical users from all over the world. A body of research provides the basis for the lexical profiling of texts. One strand of this research focuses on the development of word frequency lists based on large corpora (Coxhead, 2000; Nation, 2004, 2006a). This research, building on earlier work culminating in West’s (1953) General Service List of English Words, aims to identify the most important and useful words in the language. A second strand involves text coverage, that is the proportion of a text which is accounted for by a certain number of words. This research has found that texts of different genres vary, but the * Tel.: þ81668505897. E-mail address: [email protected]. 0346-251X/$ - see front matter Ó 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.system.2013.10.013 www.elsevier.com/locate/system Available online at www.sciencedirect.com ScienceDirect System 41 (2013) 1043e1055

Upload: dale

Post on 28-Dec-2016

221 views

Category:

Documents


2 download

TRANSCRIPT

Available online at www.sciencedirect.com

www.elsevier.com/locate/system

ScienceDirect

System 41 (2013) 1043e1055

Types of words identified as unknown by L2 learners when reading

Dale Brown*

Graduate School of Language and Culture, Osaka University, Machikaneyama-cho 1�8, Toyonaka-shi 560-0043, Japan

Received 20 February 2013; revised 23 July 2013; accepted 7 October 2013

Available online 25 October 2013

Abstract

In determining which words are likely to cause problems for learners in reading, the computer-based lexical profiling of texts hasbecome routine. This study investigates the nature of items marked as unknown by two groups of learners (n ¼ 46) when reading,with reference to the assumptions behind lexical profiling. The first assumption, that less frequent items are likely to be unknown, issupported by the results in that significantly more low frequency words were marked as unknown. The second assumption,regarding the use of the word family as the unit of counting for lexical profiling, is shown to be problematic. A significantly greaterproportion of the higher frequency words marked were found to be inflected or derived forms. The third assumption, that fewproblems stem from the fact that computers can only recognise strings of characters, may be warranted. Relatively few of the higherfrequency words that were marked occurred in the reading texts in ways likely to be unfamiliar to the participants. The study thusconcludes that in using computer-based profiling of texts to judge which words cause problems for learners, the primary issue is theuse of the word family as the unit of counting.� 2013 Elsevier Ltd. All rights reserved.

Keywords: Vocabulary; Reading; Text coverage; Lexical profiling; Word lists; Word families; Frequency effects; Word frequency

1. Introduction

In considering the types of words which cause problems for learners when reading in a second language, perhapsthe commonsense response is that words which are less frequent in the language are the primary problem. This line ofthinking can be easily operationalised today thanks to the development of lexical profiling. Lexical profiling involvesexamining the frequency level of each word in a text, allowing the less frequent items to be quickly identified. Lexicalprofiling can be conducted using freely available, simple tools such as the computer programs Range (Nation &Heatley, 2002) and AntWordProfiler (Anthony, 2013), with Cobb (2010) reporting that his online version of Range(www.lextutor.ca/vp/) has attracted pedagogical users from all over the world.

A body of research provides the basis for the lexical profiling of texts. One strand of this research focuses on thedevelopment of word frequency lists based on large corpora (Coxhead, 2000; Nation, 2004, 2006a). This research,building on earlier work culminating inWest’s (1953)General Service List of English Words, aims to identify the mostimportant and useful words in the language. A second strand involves text coverage, that is the proportion of a textwhich is accounted for by a certain number of words. This research has found that texts of different genres vary, but the

* Tel.: þ81668505897.

E-mail address: [email protected].

0346-251X/$ - see front matter � 2013 Elsevier Ltd. All rights reserved.

http://dx.doi.org/10.1016/j.system.2013.10.013

1044 D. Brown / System 41 (2013) 1043e1055

most frequent words always make up a large proportion of a text, with, for example, the top 2,000 word familiesaccounting for between 80 and 90 percent of many texts (Nation, 2006b; Webb & Rodgers, 2009a, 2009b). This workreveals how much vocabulary is required for dealing with different texts and thus assists in establishing vocabularylearning goals. A third strand looks at the amount of text coverage required for successful comprehension of texts (Hu& Nation, 2000; Laufer & Ravenhorst-Kalovski, 2010; Schmitt et al., 2011). These studies have found that high levelsof coverage are necessary, a figure of 98 percent being recommended.

Lexical profiling and the findings reported above provide teachers and materials writers with tremendously usefulinformation. Words in a text that are of lower frequency can be quickly identified, and the overall difficulty of a textestimated. These are very useful procedures, but in applying them it must be recognised that several assumptions arebeing made. This study looks at the types of words which learners themselves identify as unknown in reading texts andin doing so examines these assumptions. Viewed through the practice of lexical profiling, there are four possiblesources of difficulty for learners.

The first possibility is the central idea behind lexical profiling, that low frequency items are the main source ofdifficulty. This is based on the idea that more frequent words are learnt earlier, which Milton (2007, 2009) describes as“the frequency model of lexical learning” (2007, p. 48). The central place of frequency in language learning generallyhas been recognised in recent times (see Ellis (2002) and the responses to it), and with specific reference to vocabulary,Milton (2009) notes that Palmer (1917) had observed the frequencyedifficulty link almost a century ago. Until recentlythe idea that vocabulary learning follows frequency had been widely accepted even though there was little empiricalevidence for it. In recent years, however, the frequency model has been tested, with a number of studies (Aizawa, 2006;Brown, 2012; Milton, 2007) finding it to be robust for L2 learners in that knowledge of words is greatest at the highestfrequency level and declines at subsequent levels. Milton (2009) suggests that the frequency effect “is so powerful thatword difficulty features [such as degree of cognateness and word length], commonly accepted as influential indetermining whether or not a word will be learned, fail to significantly impact on this effect” (p. 242).

A second possibility concerns gaps in learners’ knowledge of high frequency items. As just discussed, frequency isa strong driver of learning when looking at the lexicon as a whole, but it is not the only factor and there are inevitablysome gaps in learners’ knowledge, while there are also some indications that the vocabulary knowledge of someindividual learners may not follow the frequency model (Booth, 2013; Brown, 2012; Milton, 2007). Furthermore,Browne and Culligan (2008) report that many Japanese EFL learners, the participants in this study, display consid-erable gaps in their knowledge of high frequency vocabulary in comparison with their overall vocabulary knowledge.It must also be recognised that because high frequency items comprise such a large proportion of texts, even smallgaps in learners’ knowledge of these items could mean that many words encountered in a text are unknown. Toillustrate, suppose that a text is 500 words long, and that the first 1,000 words in a frequency list make up 80 percent ofthe text, i.e. 400 of the words, while the third 1,000 make up 4 percent of the text, i.e. 20 words. A learner with a 5percent gap in their knowledge of the first 1,000 words would, on average, lack knowledge of 20 out of the 500 wordsin the text. A gap of 5 percent in their knowledge of the third 1,000 words, however, amounts to just 1 unknown word.Thus gaps in learners’ knowledge of the most frequent words could have a large impact on the overall level of vo-cabulary knowledge they bring to a text.

A third possibility arises as a result of the word family being adopted as the most appropriate unit in work on lexicalprofiling. Aword family consists of a free-standing base form (e.g. read) and a range of affixed forms, including bothinflected forms (e.g. reads, reading) and derived forms (e.g. reader, readable,misread) (Bauer & Nation, 1993). Wordfamilies can be defined more or less inclusively, that is allowing a greater or more restricted range of affixes. The wordlists featuring in much of the recent work on lexical profiling contain word families at level 6 of Bauer and Nation’sscheme. This level permits a range of affixes, but there remain affixes which are excluded and permitted affixes mustoperate in prescribed ways regarding regularity and transparency. The third possibility then is that the use of wordfamilies in the profiling of texts and estimation of difficulty may be problematic. Nation (2006b) states that “theassumption that lies behind the idea of word families is that when reading and listening, a learner who knows at leastone of the members of a family well could understand other family members by using knowledge of the most commonand regular of the English word-building devices” (p. 67). In support of this, Nation cites L1 research (Nagy et al.,1989) that shows the word family to be a psychologically real unit. Schmitt and Zimmerman’s (2002) study ofproductive knowledge of derivations also offers some support for this position in that it found that learners couldproduce derived forms for some words which they apparently did not know. This would seem to indicate that word-building processes were being applied, although it should be noted that the learners were of advanced proficiency and

1045D. Brown / System 41 (2013) 1043e1055

that their overall knowledge of derivations was deemed limited. On the other hand, Mochizuki and Aizawa (2000)have found that even learners with a considerable vocabulary size have only partial knowledge of English affixes.Ward and Chuenjundaeng (2009), similarly, have found very little evidence that Thai learners of English can make useof the word-building devices in English and thus suggest that the word family may not be the appropriate unit forlearners without knowledge of classical or Latinate languages. In research on reading, morphological awareness hasbeen shown to contribute to reading ability even when other important variables are controlled for, in both L1 and L2studies (Droop & Verhoeven, 2003; Jeon, 2011; Nagy et al., 2006; Wang et al., 2006). For example, Jeon’s study withKorean EFL learners found that, after taking into account the vocabulary size of the learners, their knowledge ofmorphology made a significant difference to their reading ability. Psycholinguistic research involving a number oflanguages and learners from varying language backgrounds shows differences between L1 and L2 morphologicalprocessing (Clahsen et al., 2010), with the results suggesting that for L2 learners the connections between members ofa word family are looser. In sum, a variety of evidence suggests that the word family assumption may be doubtful, atleast with certain types of learners. Specifically then, with its focus on Japanese learners of English, this studyconsiders the possibility that unfamiliar members of high frequency families cause problems for learners.

Finally, a fourth possibility is that unfamiliar uses of high frequency items are problematic. This possibility againarises because of the way that word lists and frequency profiles are produced. As yet computers cannot take account oftwo important facts: that most words are polysemous and are used in a variety of contexts; and that many high fre-quency items frequently occur within multi-word expressions. The assumption seems to be that these are relativelysmall problems. While most words are polysemous, the connections between senses are often quite transparent and, asParent (2012) shows, the number of homonymous words is quite small. Meanwhile, it is pointed out that there arerelatively few true idioms and that it is possible to determine the meaning of many multi-word expressions from theircomponent words (Nation & Webb, 2011). Vidal (2011), however, has found that “deceptively transparent” itemscause clear problems for learners when they are encountered in text: “in the case of false cognates, many of thesubjects failed to notice the difference in meaning between the L2 concept and the L1 one they related it to, and in thecase of polysemous words, they tended to stay with the meaning they already knew” (p. 247). With regard to multi-word items, Martinez and Murphy (2011) have found evidence of learners’ difficulties in this area. In their studylearners completed two tests, one with texts made up of highly frequent words, the other comprising texts containingthe same vocabulary items but featuring multi-word expressions made up of those items. Learners’ comprehensionscores decreased significantly on the latter, while they also tended to overestimate their understanding of the texts. Anadditional difficulty is that varied uses are often unique to each word form in a word family. Gardner (2007) haspointed out that the individual members of a word family can be individually polysemous. In the same way, multi-word items tend to involve individual members of a word family, rather than the word family as a whole, and thuscorpus linguistics focuses on the patterning of individual word forms (Sinclair, 1991, 2004). At present our word listsand frequency profiles are based on a rather nebulous concept of a word, and the appropriateness of this with respect tolearners has been little explored.

This study thus sets out to investigate the types of words that learners indicate as being problematic when reading,considering each of the four possible sources of difficulty discussed above. The four possibilities lead to the followingfour research questions:

1. Do low frequency items cause problems for learners when reading?2. Do gaps in their knowledge of high frequency items cause problems for learners when reading?3. Do unfamiliar members of high frequency word families cause problems for learners when reading?4. Do unfamiliar uses of high frequency items cause problems for learners when reading?

2. Method

2.1. Participants

The study involved two intact classes of low-intermediate level, Japanese university students. Both classes con-sisted of non-English majors from two different academic departments. A total of 51 students elected to participate inthe study initially, but since data was collected on five separate occasions, there was some attrition. Data was includedfrom participants who completed the central task on four of the five occasions, a total of 46 participants, 44 of whom

1046 D. Brown / System 41 (2013) 1043e1055

completed the task on all five occasions. The results below are adjusted to account for the two participants who missedone data collection session. The two classes from which the participants are drawn are streamed classes, so thestudents both within and between the two groups were expected to be quite similar in proficiency. This was borne outby a Yes/No vocabulary size test which showed no significant difference between the two classes.

2.2. Procedure

Two forms of data were collected. First, the participants completed a Yes/No vocabulary test. This consisted of 100items, 20 items each from the first five bands of Nation’s (2006a) BNC-based word lists. These lists contain wordfamilies compiled from the BNC, with the sequencing of the word families based on the frequency of the families inthe spoken component of the BNC. In addition, the test contained 20 pseudowords taken from Milton’s (2009) firstsample X_Lex test. The Yes/No test provides a measure of the participants’ receptive vocabulary size and an estimateof the degree of knowledge of each frequency band.

Second, on five separate occasions, at two-week intervals, the participants read short articles and were instructed asfollows: “As you read the article, circle any words that you do not know”. This instruction was given in both Englishand Japanese. The articles were from the assigned textbook for the classes, Cover to Cover 2 (Day & Harsch, 2008),thus they are designed for language learning, and are an average of 527 words in length. The articles were read as partof the participants’ normal class activities, and hence the task was preceded by lead-in activities related to the topicand followed by comprehension, summarising and/or reaction activities. While no comprehension test was given, thelearners’ successful completion of the various class activities demonstrated their broad understanding of the five texts.

The intention in using this task to discover which words cause problems for learners was to maintain a high level ofclassroom ecological validity. The desire was that the learners’ reading of the articles would take place in a fashionvery much like their usual practice, i.e. that the learners would read for comprehension rather than scan the texts forunknown items, and in their usual setting. The task was intended to be minimally intrusive into their reading, andindeed this task is often used with learning purposes in mind in L2 classrooms. It must be acknowledged, however, thatthis task is an indirect means of assessing which vocabulary items cause problems for learners; it is reliant on learnersactually indicating which words are unknown. In addition, individual learners may have differed somewhat in theirapproach to the task and in their understanding of “words that you do not know”. It will also be noted that the in-struction did not offer a definition of “words”. The participants, however, overwhelmingly marked single orthographicwords in the articles. On 12 occasions a multi-word item of some kind was marked, almost all phrasal verbs such aslive up to, end up and set off, as compared with 1,088 single orthographic words marked. These 12 items wereexcluded from the results, there being no simple way of including them given the nature of the frequency lists the studymakes use of.

On analysing the words marked as unknown, it became clear that the participants had been unsure whether anunknown word appearing more than once in the same article should be marked each time it appeared. Most, but not all,participants marked such words only once. Hence, for the small number of participants who marked such words eachtime they appeared in an article, the word is only counted once. For the same reason, the focus in the analyses below ison the number of types occurring in the texts rather than tokens.

3. Results and discussion

3.1. Yes/No test results

The Yes/No vocabulary test was scored in the simple manner recommended by Milton (2009) of multiplying eachword checked by 50 (since each word on the test represents 50 word families on the list) and subtracting 250 for eachpseudoword checked. This found a mean vocabulary size of 3,461 (SD¼ 343). An average of 1.43 pseudowords werechecked per participant, for a false alarm rate of 7.15 percent, while the test had good reliability (Cronbach’sa¼ 0.82). Table 1 shows the results for each frequency band, without adjustment for pseudowords checked. As can beseen, the participants indicated almost full knowledge of the first thousand word families, and in fact a large majorityof the participants indicated that they knew all 20 items in this band.

1047D. Brown / System 41 (2013) 1043e1055

3.2. Frequency

Research questions 1 and 2 are similar in that they ask where along the scale of frequency learners’ problems lie. Anestimate of where this might be can be made with the measure of the proportion of words known in each frequencyband and a vocabulary profile of the articles. The texts were profiled with the Range (Nation & Heatley, 2002)program, again using Nation’s (2006a) word lists. To estimate the number of words that may be unknown, the pro-portion of items unknown at each frequency level as indicated by the Yes/No test (unadjusted for pseudowordschecked) is multiplied by the number of types from that level contained in the articles. For example, the participantsindicated that 85.85 percent of the K2 words are known and the text profile shows that there are 142 different K2 typesin the articles. Thus it may be predicted that the participants know 85.85 percent� 142¼ 122 of the K2 types. In turn,this implies an estimated 20 (142 � 122) of the K2 words will be unknown to these participants. That is, eachparticipant might be expected to mark as unknown 20 K2 words across the five texts.

It seems likely, of course, that the participants would mark fewer words than this. First, the calculations use the Yes/No results without adjustment for the number of pseudowords marked. Second, the Yes/No test presents the words inisolation and the participants’ attention is focused solely on judging whether each word is known or not. Whenmarking the words in the articles as unknown, in contrast, the words are seen in context and the participants are alsoreading for meaning at the same time. The context may allow an unknown word to be instantly inferred and so it maygo unmarked, or it may allow the recognition of a word that is partially known or that is in a process of attrition.Participants may also simply pass over words that are wholly unknown because they are focused on building meaningfrom the text. Indeed, two previous studies (Laufer & Yano, 2001; Wan-a-rom, 2010) have compared words indicatedas known in a reading text and learners’ subsequent abilities to demonstrate knowledge of the meaning of those words.These studies indicate that learners tend to overestimate their vocabulary knowledge to some extent, and it is likelythat in this study also participants did not mark some words which are in fact unknown to them. With these caveats inmind, Table 2 nevertheless presents the estimates.

It is interesting to compare the predicted numbers with those actually found (Fig. 1). These numbers are for the 46participants as a whole and have been adjusted for the fact that two participants missed one of the five sessions whenthe articles were presented. Clearly, the estimates are a great deal higher than the actual number of words markedacross all the frequency levels. This would suggest that marking words as unknown in a text does indeed lead learnersto underreport which words are unknown, while in addition the Yes/No results may reflect some overreporting of theparticipants’ vocabulary knowledge. Nevertheless, the seeming underreporting is not uniform across the frequencybands. If we consider the percentage of the estimate that was in fact marked as unknown, there are differences inaccordance with frequency. For K1eK3, there are very large gaps between the estimate and the actual, with only 20percent, 16 percent and 26 percent respectively of the estimated number actually marked as unknown. This may meanthat quite a number of these words are known to some extent, and are more easily recognised in context through theprocesses described earlier. For K4 and K5, the number of words marked unknown is closer to the estimate, repre-senting 52 percent and 46 percent of it respectively. This may be a sign that at these levels the learners are encounteringwords that are wholly unknown to them.

Also of note in Fig. 1 is that a small number of proper nouns were marked as unknown. Proper nouns are widelyassumed to be unproblematic for learners when reading, certainly for learners beyond the most elementary proficiencylevels, though this is an assumption that I have questioned previously (Brown, 2010). The five articles actually containa relatively small number of proper nouns, and the proper nouns marked by participants as unknown are generallyunremarkable in that each begins with a capital letter. While proper nouns are clearly not a major problem for theselearners, they are not wholly unproblematic either.

Table 1

Results for the Yes/No test by frequency band (N ¼ 46).

Mean SD Range

1K 19.63 0.645 18e202K 17.17 1.691 11e20

3K 14.83 1.992 8e19

4K 14.89 2.470 7e19

5K 9.87 2.986 4e17

Table 2

Estimates for the number of words unknown to participants in the articles.

A: Proportion of words

indicated as known

on the Yes/No test

B: Number of

types appearing

in the articles

C: Estimate of number of types in

the articles known (A � B)

D: Estimate of number of types

in the articles unknown (B � C)

1K 98.15% 567 557 10

2K 85.85% 142 122 20

3K 74.15% 58 43 15

4K 74.45% 26 19 7

5K 49.35% 20 10 10

Other 50%a 36 18 18

Proper nouns 100%b 20 20 0

a This figure does not come from the Yes/No test; rather it is based on the assumption that the participants’ knowledge of these words will be

approximately the same as their knowledge of the K5 words.b This figure likewise does not come from the Yes/No test. Instead, it reflects the widespread assumption that proper nouns are unproblematic for

learners.

1048 D. Brown / System 41 (2013) 1043e1055

Focusing now on the actual number of words marked as unknown, there is a general trend towards greater problemswith less frequent words, and the largest number of words marked is beyond the 5K level. Since the data are notnormally distributed, Friedman’s ANOVA was used and confirms that there is a significant difference between thenumber of words marked in these six categories (i.e. the category of proper nouns was not included in this analysis): x2

(5) ¼ 72.194, p < .001. Post hoc Wilcoxon signed rank tests were then conducted to compare the number of wordsmarked at each frequency band with the successive band. As five comparisons were made a Bonferroni correction wasapplied, adjusting the alpha from .05 to .01. The post hoc tests found a significant difference between the 1K and 2Klevels, 2K and 3K and 4K and 5K, while the other two comparisons were not found to be significant.1

It should be noted that the trend towards more words being marked unknown from lower frequency levels occursdespite the fact that the number of words from each band markedly decreases (see Table 3). As shown, the proportionof items that were marked as unknown increases greatly in the lower frequency bands.

Also noteworthy is that the data suggest that any gaps that do exist in the learners’ knowledge of high frequencyitems are not generalised for this group of participants. This is shown by examining the mean number of participantswho marked a particular item. Some items were marked as unknown by just a single participant, while others weremarked by large numbers of participants. As Table 4 shows, there were clear differences across the frequency bands.At the higher frequency levels there tended to be a great deal of diversity among the participants as to which wordswere marked. This would appear to indicate that any gaps in the participants’ knowledge of higher frequency wordsare on the whole idiosyncratic rather than generalised. At the lower frequency levels, however, it seems that manyparticipants marked the same words as unknown, indicating more general problems at these levels.

3.3. Word families

The results in section 3.2 show that the first possibility, that low-frequency items cause the majority of problems forlearners when reading, may be correct. Nevertheless, high frequency words were marked as unknown by the par-ticipants, and the third and fourth possibilities considered in section 1 seek to understand possible reasons for this.Research question 3 thus asks whether unfamiliar members of high frequency word families cause problems forlearners. The 2,000 word families in the K1 and K2 bands of Nation’s (2006a) word lists consist of 11,941 word forms,an average of just under six per family. In using word families as the unit for calculating text coverage and receptivevocabulary size, the assumption, as mentioned earlier, is not that each and every word form will be known, but thatwhen encountering unfamiliar forms in context, learners are able to deal with the item using their knowledge ofinflectional and derivational morphology.

To consider this issue, each item marked as unknown by the participants was checked in Nation’s (2006a) lists as towhether the item was the headword of the family, an inflected form or a derived form: e.g. act is a headword, admitted

1 Due to an error in data collection it was not possible on one occasion to identify which of 19 participants had marked words as unknown. This

analysis is then based on a subset of 209 out of the 228 articles for which the identity of participants is known.

Fig. 1. Estimated and actual number of items marked as unknown.

1049D. Brown / System 41 (2013) 1043e1055

is an inflected form (admit being the headword) and advertisers is a derived form (advertise being the headword).Table 5 shows the results. Looking at the proportions of the three categories, there appears to be a difference betweenthe more frequent bands (K1eK3) and the less frequent (K4eK5). In the more frequent bands, inflected and derivedforms make up the majority of the words marked, while in the less frequent bands headwords account for the largestproportion. A Pearson’s chi-square analysis was thus conducted comparing the number of headwords, inflected formsand derived forms across the frequency levels. This analysis shows a significant association between frequency bandand word type: c2 (8)¼ 126.221, p< .001. Examining the standardized residuals reveals that there were significantlyfewer headwords marked as compared to the overall figures at the 1K, 2K and 3K levels and significantly moreheadwords at the 4K level. There were significantly more inflected forms marked at the 2K level and significantlyfewer at the 5K level. Finally, there were significantly more derived forms marked at the 1K and 3K levels andsignificantly fewer at the 2K and 4K levels. A second analysis comparing the number of headwords versus the numberof inflected and derived forms combined shows these trends more clearly. Again the overall association betweenfrequency band and word type is significant: c2 (4) ¼ 78.952, p < .001. The standardized residuals reveal the sameresults as above of course for the number of headwords, while for the combined number of inflected and derived forms,there are significantly more at the 1K and 2K levels, and significantly fewer at the 4K level. These results then showthat at higher frequency levels headwords are less often unknown, but there appear to be gaps in the learners’knowledge of other forms of word families. On the other hand, at less frequent levels entire families appear to beunknown and hence whether the item is a headword, inflected form or derived form does not affect whether it ismarked or not.

Inflected and derived forms of high frequency word families do then seem to be one source of difficulty for learners.However, it also seems that the frequency of a word form itself is an important factor. Evidence for this can be seen inTable 6. This table contrasts the frequency in the Corpus of Contemporary American English (COCA)(Davies, 2008)of derived forms from the first three frequency bands which at least 10 percent of the participants marked as unknownwith the frequency of derived forms which no participants marked. Clearly, there are large differences between thefrequency bands, and the great differences in the number of items in each category makes statistical tests impossible.

Table 3

Total types marked as unknown and total types viewed.

A: Total number of types

marked as unknown

B: Total number of types

viewed by all 46 participants

A as percentage of B

K1 91 25,928 0.4

K2 148 6480 2.3

K3 177 2644 6.7

K4 166 1178 14.1

K5 208 910 22.9

Other 298 1642 18.1

Table 4

Mean number of times an item was marked as unknown by participants.

Mean times an item was marked as unknown

K1 2.8

K2 2.9

K3 5.5

K4 11.9

K5 17.3

Other 13.5

1050 D. Brown / System 41 (2013) 1043e1055

Nevertheless, there does seem to be a striking difference within each frequency band between the commonly markedand universally unmarked items. This suggests that the application of word-building knowledge may have a limitedrole and instead the frequency of individual word forms themselves may determine whether they are problematic forlearners or not.

3.4. Unfamiliar uses

The fourth research question asked if unfamiliar uses of high frequency items may cause problems for learners. It ispossible that the methodology employed in this study does not allow this issue to be fully addressed, since whenencountering a familiar word form which is being used in an unfamiliar way, learners may not recognise that this is soand may assign the known meaning to the form. As discussed in section 1, Vidal (2011) and Martinez and Murphy(2011) have found evidence of this with regard to polysemy and multi-word items respectively.

Despite this difficulty, it was felt that it is worthwhile to explore whether the words marked as unknown are used inthe articles in an unfamiliar way. Determining whether a use is familiar or not is of course problematic. The approachtaken was to assume that the most typical uses of items are known while less typical uses are not. Specifically, Iexamined the articles and the way in which each K1, K2 and K3 item identified by participants as unknown is actuallyused therein. The analysis was confined to these levels since on the whole the participants displayed good knowledgeof the levels and I wished to explore why these specific words from these levels had been marked. In assessing the useof the items in the reading texts, I attempted to identify anything which could potentially have caused the problem forthe participants. This meant considering the sense in which a word was used, whether a word was part of a multi-worditem, whether inflected and derived forms could be understood compositionally by applying the basic meaning of theheadword plus the inflectional/derivational affix or whether the word was used in an unusual context. This analysisdepended largely on intuition, though dictionaries (the Cambridge Advanced Learner’s Dictionary and LongmanDictionary of Contemporary English) and corpora (COCA) were also referred to. Table 7 presents some examplesof items classified as occurring in a less typical way.

As Table 8 reveals, the number of items marked as unknown which occurred in the articles in some sort of lesstypical way is not especially large. Nevertheless, it is likely that this factor does play some part in learners’ difficultieswhen reading. As discussed before, it is likely that the complete picture of the difficulties caused by words used inunfamiliar ways is not shown here due to the methodology used.

3.5. Combinations of factors

The previous two sections have examined two possible reasons why highly frequent words were marked as un-known by the participants. As mentioned in section 1, sometimes these two factors overlap: armed in armed robbery

Table 5

Number of headwords, inflected forms and derived forms among items marked as unknown by participants.

K1 (91) K2 (148) K3 (177) K4 (166) K5 (208)

Count % Count % Count % Count % Count %

Head 21 23 45 30 57 32 113 68 103 50

Inflection 30 33 74 50 50 28 37 22 33 16

Derivation 40 44 29 20 70 40 16 10 72 35

Table 6

Mean frequency in COCA (per million words) of derived forms marked by at least 10 percent of participants and derived forms unmarked by any

participants.

Marked by at least 10 percent of participants

(Mean frequency per million words)

Unmarked by any participants

(Mean frequency per million words)

K1 (n ¼ 2) 10 K1 (n ¼ 40) 131

K2 (n ¼ 1) 7 K2 (n ¼ 8) 38

K3 (n ¼ 4) 5 K3 (n ¼ 2) 14

1051D. Brown / System 41 (2013) 1043e1055

does not just have a different sense from the familiar body part meaning, it is also an inflected form of the headword.Sometimes too neither process is at work, that is words were marked which are headwords in their family and wereused in what is assumed to be a familiar way. Table 9 lists such words from the three highest frequency bands, with theadditional condition that more than one participant marked each word as unknown. These lists are clearly rather short.Nevertheless, it is worth considering what it is about these words that may have led to them being marked. One strikingthing about them is that, particularly the K1 and K2 items, they seem for the most part to belong to rather formalregisters. Corpus data backs up this assertion. Searches were conducted in COCA (Davies, 2008) to determine therelative frequency of these words in COCA’s five primary sub-divisions. Five of the nine K1 and K2 words (current,indeed, specific, cope and demonstrate) are most frequent in the academic division, each being approximately two tothree times as frequent in academic discourse as compared with their overall frequency. There seems to be somecorrelation here with Horst’s (2010) study of L2 English classrooms in Canada, which found that among the K1 wordsnot appearing in a teacher’s speech were words related to topics such as business and government and also words thatare “more characteristic of writing than speech” (p. 173). Clearly, the Japanese classroom context is likely to differsomewhat, and these participants’ main sources of input may in fact have been written rather than oral texts, and yetHorst’s comment does seem apt for these K1 and K2 items. The K3 items are less marked, most being neitherrelatively frequent nor infrequent in any of the COCA sub-divisions. Given that the Yes/No vocabulary test suggestedthat around a quarter of K3 words are unknown to the participants, these items may simply fall into that group.

A final perspective on the characteristics of the words marked as unknown by the learners can be gained byexamining the most frequently marked items. Table 10 gives the 16 words that were marked by half or more of theparticipants. Of these, 13 are low frequency (bold), defining low frequency as from K4 or lower bands, 9 are inflectedor derived members of the word family (italics), including all 3 items from the upper frequency bands, and 3 are usedin the articles in a less typical way (underlined). All 16 items are thus characterised in one of these three ways, and theprevalence of the three characteristics among these words is roughly proportional to their prevalence among the itemsmarked as unknown by learners as a whole. This is suggestive of the reasons for the selection of these items by a largenumber of the participants and these items provide a summary of the features discussed in the earlier sections.

4. Discussion

The results above provide some insights into the validity of the corpus-based development of frequency lists andtheir application to language teaching in terms of text profiling. The practice of lexical profiling assumes that the typeof words that cause learners difficulty is items of low frequency and research question 1 asked whether this is the case.On the whole, this assumption appears to be sound, as demonstrated by the significantly greater number of itemsmarked from the lower frequency levels. The largest number of items that participants marked as unknown were those

Table 7

Examples of items classified as occurring in a less typical way.

Item Use in the reading Comments

title light Heavyweight title Participants may be familiar with its naming sense rather than this sporting sense.

inch inched closer to closer Participants may be more familiar with it as a noun rather than this verbal use.

arm armed robbery Participants may be unfamiliar with it meaning weapon, additionally it occurs here as a participle adjective.

earn earned her a chance to Participants may associate the item only with money.

upset upset stomach Participants may be more familiar with it meaning worried or unhappy.

act this simple act can Participants may be more familiar with act as a verb with action as its corresponding noun form.

Table 8

Number of items used in typical and less typical ways among items marked as unknown by participants.

K1 (91) K2 (148) K3 (177)

Count % Count % Count %

Typical use 71 78 92 62 150 85

Less typical use 20 22 56 38 27 15

1052 D. Brown / System 41 (2013) 1043e1055

beyond the 5K level, and these items were marked by a large proportion of the participants. Indeed, only ten itemsbeyond the 5K level were unmarked by any participant, and nine of these are loanwords in Japanese (e.g. Halloween,yoga), simple inflections of loanwords (penguins, pandas) or partial loanwords (heavyweight).

Research question 2 asked whether gaps in learners’ knowledge of high frequency items could be a problem. Itseems that the answer is essentially no. The most frequent vocabulary is largely unproblematic for learners and gaps inlearners’ knowledge of these items are mostly specific to individual learners. Any possibly more generalised gaps thatdo exist may be predominantly in more formal, written language. It may be that learners, whether learning for ac-ademic purposes or not, require greater exposure to these sorts of texts in order to develop a well-rounded vocabulary.The dominance in textbooks of what may be termed lifestyle topics could be part of the problem and it may berecommended that language classrooms need to feature a richer variety of text types.

Third, the study considered whether the word family is the most appropriate unit for lexical profiling (researchquestion 3). This assumption is more questionable. At the higher frequency levels significantly more of the wordsmarked by participants were inflected or derived forms. Such word forms do then seem to be a source of difficulty forlearners. This study thus backs up research (Mochizuki & Aizawa, 2000; Ward & Chuenjundaeng, 2009) showing thateven learners with a relatively large vocabulary have limited knowledge of derivational affixes, and the suggestion thatuntil learners reach quite advanced levels it is likely that their ability to make use of word-building knowledge and dealwith word families will be limited. On a more positive note, those family members that are relatively frequent in theirown right will, it seems, be encountered and learnt in turn.

Fourth, regarding the final research question, the study examined whether multiple uses of a single word form, interms of polysemy and the occurrence of words in multi-word units, is a source of difficulty for learners. The datasuggest that this is not a major problem, though the limitations of the methodology with respect to this particular issuemust be borne in mind. It is possible that learners routinely fail to recognise that a familiar word is being used in anunfamiliar way.

In sum, when considering the types of vocabulary that cause problems for learners when reading, the use of corpus-based frequency lists and lexical profiling seems primarily questionable with respect to the use of the word family asthe basic unit. It does not seem to be a simple matter for learners to make use of word-building processes in order todeal with the members of a word family when reading. What then are the alternatives to the word family? One oft-mentioned alternative is the lemma: “a set of lexical forms having the same stem and belonging to the same majorword class, differing only in inflection and/or spelling” (Francis & Ku�cera, 1982, p. 1). Milton (2009) recommendslemmas as the most appropriate unit for all bar very beginners and those at very advanced levels. Nation and Webb(2011) suggest that lemmas (or possibly word types) might be the best unit when considering productive rather than

Table 9

Items marked by more than one participant which are the headwords of word families and which were

used in the articles in what is assumed to be a familiar way.

K1 K2 K3

current combine admire

forward cope arrest

indeed demonstrate capture

specific entire ease

prospect foundation

intense

saint

sore

thief

Table 10

Items marked as unknown by half or more of the partic-

ipants. Bold indicates items from frequency band K4 or

below. Italics are used for inflected or derived members of

word families. Underlining shows items used in a less

typical way in the articles.

hatred

inched

inconceivably

grinned

headquarters

portray

immune

reinforcement

unanimously

vulnerable

rigorous

brag

repertoire

defiant

enthralled

masterminds

1053D. Brown / System 41 (2013) 1043e1055

receptive language. Gardner (2007) too discusses lemmas as one possible alternative to word families. The resultsreported above might, however, be seen as suggesting that lemmas also are problematic, given that Table 5 showsinflected forms of high frequency word families were also frequently marked as unknown by the learners. This maynot though be the case. These results are based on inflected forms and these forms are not necessarily inflections of theheadword of the family. This is because the headword of a family is not the head of a lemma; it is simply a word form.For example, inched is classed as an inflected form, but is an inflection of the lemma inch (v.), rather than of inch (n.)which is the dominant lemma in the family. Indeed, these inflected forms may be lemmas in their own right, forexample, in the case of participle adjectives. Aword family is not simply a lemma with derivations added; the lemmais a fundamentally different unit. The lemmamay be a viable alternative toword families, but only further research canconfirm or refute this.

5. Conclusions

This study has sought to discover the types of vocabulary that cause problems for learners when reading. There areclearly several limitations to the study. First, the participants were working with texts that have been written spe-cifically for learners. These texts have thus been graded and are at a level appropriate to these learners, as shown bytheir ability to comprehend the texts and other similar texts in the coursebook throughout the course. The study doesthen have the advantage of ecological validity with regard to typical work conducted in ELT reading classrooms. It islikely, however, that learners may face problems of a somewhat different nature when encountering ungraded textsoutside the classroom. Second, the primary task, having participants mark unknown words in the texts as they read, isan indirect means of accessing the problems encountered by learners when reading. This task does have some pedigreein research on vocabulary and reading (e.g. Carver, 1994; Laufer, 1989), and its adoption in this study had theadvantage of enabling a larger number of participants to be involved and for each participant to consider a greaterquantity of text, the participants reading a total of over 2600 tokens in the five articles as they completed the task.Nevertheless, the results are dependent on the learners’ indications of problematic words. An online method, forexample, using eye tracking, may have provided a different picture of the difficulties with vocabulary that learnershave. The indirect nature of the task also means that individual learners may have been operating with differentdefinitions of “words that you do not know”. In particular, one explanation for the marking of proper nouns as un-known may be that learners had never encountered them before as opposed to not recognising them as proper nouns.2

2 I would like to thank Barry Lee Reynolds for making this suggestion.

1054 D. Brown / System 41 (2013) 1043e1055

Despite these limitations, the study does suggest that there are some problems with the assumptions that lie behindthe computer-based lexical profiling of texts. The central assumption that less frequent words are likely to be unknownis supported. However, the use of word families as the normal unit in lexical profiling seems more questionable. Thereis a need for further work critically examining this issue and the extent to which the units used match learners’psycholinguistic reality. This work must take account of factors such as varying L1 backgrounds, age and L2 pro-ficiency. Likewise, while the results did not reveal major problems with regard to unfamiliar uses of frequent words, itseems sensible to recommend further research in this area too with regard to the impact unfamiliar uses have onlearners’ reading, and how this factor interacts with the issue of word families. Lexical profiling is a very usefulprocedure, and in order that teachers and learners may best benefit from it, we must strive to ensure that the proceduresused are sound, not just computationally convenient.

Acknowledgements

I would like to thank Barry Lee Reynolds, Phil Bennett and the three anonymous reviewers for their comments onprevious versions of this paper.

References

Aizawa, K., 2006. Rethinking frequency markers for English-Japanese dictionaries. In: Ishikawa, S., Minamide, K., Murata, M., Tono, Y. (Eds.),

English Lexicography in Japan. Taishukan Publishing Company, Tokyo, pp. 108e119.

Anthony, L., 2013. AntWordProfiler. Waseda University, Tokyo, Japan.

Bauer, L., Nation, P., 1993. Word families. International Journal of Lexicography 6, 253e279.

Booth, P., 2013. Vocabulary knowledge in relation to memory and analysis: An approximate replication of Milton’s (2007) study on lexical

profiles and learning style. Language Teaching 46, 335e354.

Brown, D., 2012. The frequency model of vocabulary learning and Japanese learners. Vocabulary Learning and Instruction 1, 20e28.Browne, C., Culligan, B., 2008. Combining technology and IRT testing to build student knowledge of high frequency vocabulary. The JALT

CALL Journal 4, 3e16.

Brown, D., 2010. An improper assumption? The treatment of proper nouns in text coverage counts. Reading in a Foreign Language 22, 355e361.

Cambridge Advanced Learner’s Dictionary, 2008. Cambridge University Press, Cambridge.

Carver, R.P., 1994. Percentage of unknown vocabulary words in text as a function of the relative difficulty of the text: Implications for instruction.

Journal of Literacy Research 26, 413e437.

Clahsen, H., Felser, C., Neubauer, K., Sato, M., Silva, R., 2010. Morphological structure in native and nonnative language processing. Language

Learning 60, 21e43.

Cobb, T., 2010. Learning about language and learners from computer programs. Reading in a Foreign Language 22, 181e200.

Coxhead, A., 2000. A new academic word list. TESOL Quarterly 34, 213e238.

Davies, M., 2008. The Corpus of Contemporary American English (COCA): 400þ million words, 1990-present.

Day, R.R., Harsch, K., 2008. Cover to Cover 2. Oxford University Press, New York.

Droop, M., Verhoeven, L., 2003. Language proficiency and reading ability in first- and second-language learners. Reading Research Quarterly 38,

78e103.

Ellis, N.C., 2002. Frequency effects in language processing. Studies in Second Language Acquisition 24, 143e188.Francis, W., Ku�cera, H., 1982. Frequency Analysis of English Usage. Houghton Mifflin, Boston.

Gardner, D., 2007. Validating the construct of word in applied corpus-based vocabulary research: A critical survey. Applied Linguistics 28,

241e265.Horst, M., 2010. How well does teacher talk support incidental vocabulary acquisition? Reading in a Foreign Language 22, 161e180.

Hu, M., Nation, I.S.P., 2000. Unknown vocabulary density and reading comprehension. Reading in a Foreign Language 13, 403e430.

Jeon, E.H., 2011. Contribution of morphological awareness to second-language reading comprehension. The Modern Language Journal 95,

217e235.Laufer, B., 1989. What percentage of text-lexis is essential for comprehension? In: Lauren, C., Nordman, N. (Eds.), Special Language: From

Humans Thinking to Thinking Machines. Multilingual Matters, Clevedon, pp. 316e323.

Laufer, B., Ravenhorst-Kalovski, G.C., 2010. Lexical threshold revisited: Lexical text coverage, learners’ vocabulary size and reading compre-

hension. Reading in a Foreign Language 22, 15e30.Laufer, B., Yano, Y., 2001. Understanding unfamiliar words in a text: Do L2 learners understand how much they don’t understand? Reading in a

Foreign Language 13, 549e566.

Longman Dictionary of Contemporary English, 2009. Longman, Harlow.

Martinez, R., Murphy, V.A., 2011. Effect of frequency and idiomaticity on second language reading comprehension. TESOL Quarterly 45,

267e290.

Milton, J., 2007. Lexical profiles, learning styles and the construct validity of lexical size tests. In: Daller, H., Milton, J., Treffers-Daller, J. (Eds.),

Modelling and Assessing Vocabulary Knowledge. Cambridge University Press, Cambridge, pp. 47e58.

1055D. Brown / System 41 (2013) 1043e1055

Milton, J., 2009. Measuring Second Language Vocabulary Acquisition. Multilingual Matters, Bristol.

Mochizuki, M., Aizawa, K., 2000. An affix acquisition order for EFL learners: An exploratory study. System 28, 291e304.

Nagy, W., Anderson, R.C., Schommer, M., Scott, J.A., Stallman, A.C., 1989. Morphological families in the internal lexicon. Reading Research

Quarterly, 262e282.

Nagy, W., Berninger, V.W., Abbott, R.D., 2006. Contributions of morphology beyond phonology to literacy outcomes of upper elementary and

middle-school students. Journal of Educational Psychology 98, 134e147.

Nation, I.S.P., 2004. A study of the most frequent word families in the British National Corpus. In: Bogaards, P., Laufer, B. (Eds.), Vocabulary in a

Second Language, Language Learning and Language Teaching. John Benjamins Publishing Company, Amsterdam, pp. 3e13.

Nation, I.S.P., 2006a. BNC-based word lists. Victoria University of Wellington, Wellington.

Nation, I.S.P., 2006b. How large a vocabulary is needed for reading and listening? The Canadian Modern Language Review/La revue canadienne

des langues vivantes 63, 59e81.

Nation, I.S.P., Heatley, A., 2002. Range: A Program for the Analysis of Vocabulary in Texts.

Nation, I.S.P., Webb, S., 2011. Researching and Analyzing Vocabulary. Heinle Cengage Learning, Boston.

Palmer, H.E., 1917. The Scientific Study and Teaching of Languages. Harrap, London.

Parent, K., 2012. The most frequent English homonyms. RELC Journal 43, 69e81.

Schmitt, N., Jiang, X., Grabe, W., 2011. The percentage of words known in a text and reading comprehension. The Modern Language Journal 95,

26e43.

Schmitt, N., Zimmerman, C.B., 2002. Derivative word forms: What do learners know? TESOL Quarterly 36, 145e171.Sinclair, J., 1991. Corpus, Concordance, Collocation. Oxford University Press, Oxford.

Sinclair, J., 2004. Trust the Text: Language, Corpus and Discourse. Routledge, London.

Vidal, K., 2011. A comparison of the effects of reading and listening on incidental vocabulary acquisition. Language Learning 61, 219e258.Wan-a-rom, U., 2010. Self-assessment of word knowledge with graded readers: A preliminary study. Reading in a Foreign Language 22,

323e338.

Wang, M., Cheng, C., Chen, S., 2006. Contribution of morphological awareness to Chinese-English biliteracy acquisition. Journal of Educational

Psychology 98, 542.

Ward, J., Chuenjundaeng, J., 2009. Suffix knowledge: Acquisition and applications. System 37, 461e469.

Webb, S., Rodgers, M.P.H., 2009a. The lexical coverage of movies. Applied Linguistics 30, 407e427.

Webb, S., Rodgers, M.P.H., 2009b. Vocabulary demands of television programs. Language Learning 59, 335e366.

West, M., 1953. A General Service List of English Words. Longman, Green and Co, London.