Testing reading comprehension

Evaluation and Assessment in Language Education

Dr Kia Karavas

The reading process

Reading is a dynamic process in which the reader interacts with the text to construct meaning.  Inherent in constructing meaning is the reader's ability to activate prior knowledge, use reading strategies and adapt to the reading situation.

What can a skilled reader do?

A skilled reader rapidly and accurately decodes the words, attaches the meaning to words and sentences, connects text information to relevant background knowledge, maintains a mental representation of what he or she has already read, forms hypotheses about upcoming information and makes decisions based on his or her purpose for reading – all at the same time. Carlisle and Rice, 2002

What influences reading difficulty?

• Reader variables The reader’s background and subject/topic knowledge,

their cultural knowledge and their knowledge of the language in which the target texts are written. The reader’s ability to process printed information is clearly crucial.

• Text variables Linguistic features of text clearly affect readability of text

and reader’s comprehension, and text type, organization and so on as well as text topic influence how well readers can process meaning. Many features of text need to be considered in the design of tests of reading.

What should assessment of reading focus on?

• Alderson argues that we should focus our assessment of reading on the target skills we want out students to develop.

Major reading subskills include• Reading quickly to skim for gist, scan for specific details,

and establish overall organisation of the text• Reading carefully for main ideas, supporting details,

author’s argument and purpose, relationship of paragraphs, and fact vs opinion

• Drawing inferences from both stated and implied content

Enabling reading skills include

• Understanding at the sentence level (lexis, syntax, cohesive markers)

• Understanding at inter-sentence level (e.g. anaphoric/catophoric reference, recognizing discourse markers)

• Understanding components of non-linear texts (meaning of graph, chart labels etc)

Choosing texts

• Many sources for reading texts: purpose written, taken directly from authentic material or adapted.

• Use a variety of text types; do not select texts of a particular kind just because they are readily available

• Choose texts of appropriate length; detailed reading can be assessed with texts consisting of a few sentences. Texts of up to 2000 words may be used for extensive reading at higher levels

•

Choosing texts

• Choose texts with many discrete pieces of information• Choose texts that will interest the candidates but will not disturb or

excite them. Avoid texts with controversial or biased material (e.g. abortion, international disputes, religion). These are not suitable for assessment because they can upset candidates and affect the reliability of results.

• You should check the language of your reading texts. Ninety percent (90%) of the words in a text should be known to students for good comprehension (Nation 1990).

• Choose topics within the experience of the test takers. However, avoid texts that contain information which is part of the test takers’

general background knowledge.

Developing test items/questions

• The wording of the reading test questions should not cause comprehension difficulties to learners. It should always be within their capabilities and less demanding than the text itself.

• Responses to test items should make minimal demands on writing ability.

• The items should be in the same order as the information in the text. Mixing up the order of questions increases the difficulty level.

Assessing reading comprehension: Three levels

• The first level, literal comprehension, is the most obvious. Comprehension at this level involves surface meanings. At this level, teachers can ask students to find information and ideas that are explicitly stated in the text.

Levels of comprehension

• The second level is interpretive or referential comprehension. At this level, students go beyond what is said and read for deeper meanings. They must be able to read critically and analyse carefully what they have read. Students need to be able to see relationships among ideas, for example, how ideas go together and also see the implied meanings of these ideas. It is also obvious that before our students can do this, they have to first understand the ideas that are stated (literal comprehension).

Referential Level of comprehension

• At this level, teachers can ask more challenging questions such as asking students to do the following:– Re-arrange the ideas or topics discussed in the text.– Explain the author's purpose of writing the text.– Summarize the main idea when this is not explicitly

stated in the text.– Select conclusions which can be deduced from the

text they have read.

Levels of comprehension

• Finally, the third level of comprehension is critical reading whereby ideas and information are evaluated. Critical evaluation occurs only after our students have understood the ideas and information that the writer has presented. At this level, students can be tested on the following skills:– The ability to differentiate between facts and opinions.– The ability to recognize persuasive statements .– The ability to judge the accuracy of the information

given in the text.

Some tips on item writing

• Do not write items for which the correct response can be found without understanding the text.

• Do not write items that some candidates are likely to answer from general knowledge without reading the text.

• Make the items independent of each other; do not make the correct response on one item depend on another item being responded to correctly.

Possible testing formats for reading tests: Multiple choice

• Multiple-choice questions are a common device for testing students’ text comprehension. They allow testers to control the range of possible answers to comprehension questions, and to some extent to control students’ thought processes when responding.

An example

A. is written in an academic register.B. has an impersonal tone.C. is written as a narrative.

1.1 Read the first part of the text on p. 2 (the left side column), and choose the best answers (A, B, or C) for items 1-4.1. This text

A. argue for better housing for poor people.B. describe the writer’s family life.C. explain what is needed to become a writer.

2. One aim of this text is to

A Publishing there is of a high standard.

B Writers have achieved more freedom there.

C Publishing still has a long way to go there.

8. What does the writer say about North Africa?

Matching

• One alternative objective technique for the testing of reading is multiple matching. Here two sets of stimuli have to be matched against each other as, for example, matching headings for paragraphs to their corresponding paragraph, titles of books against extracts from each book, and so on

Example

A. Rules for playing a game B. Written safety instructionsC. Instructions for paying tax D. Reverse side of a book’s title pageE. University brochure F. Travel guide

21. Exception for children under age 18. If you are planning to claim a return for your child, who was under 18 at the end of 2009, and certain other conditions apply, you can include your child’s income on your form.

22. All rights reserved. No part of this publication may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or any other information storage or retrieval system, without permis sion in writing from the publishers.

23. The inn is a little oasis amidst the narrow and bustling cobbled streets at the heart of the historic city.

Read the following extracts (21-25) and decide in which publication they might appear. Use each of the options below (A-H) only once. There is one option you do not need.

Ordering tasks (Strip stories)

• Candidates are given a scrambled set of words, sentences, paragraphs or texts and have to put them into their correct order. The ordering tasks can offer the possibility of testing the ability to detect cohesion, overall text organization.

Short-answer tests

• Test-takers are simply asked a question which requires a brief response, in a few words, not just Yes/No or True/False. The justification for this format is that it is possible to interpret students’ responses to see if they have really understood, whereas on multiple-choice items students give no justification for the answer they have selected and may have chosen one by eliminating others.

The free-recall test

• In free-recall tests (sometimes called immediate-recall tests), students are asked to read a text, to put it to one side, and then to write down everything they can remember from the text. The free-recall test is often held to provide a purer measure of comprehension, since test questions do not intervene between the reader and the text.

The summary test

• Students read a text and then are required to summarize the main ideas, either of the whole text or of a part, or those ideas in the text that deal with a given topic. It is believed that students need to understand the main ideas of the texts, to separate relevant from irrelevant ideas, to organize their thoughts about the text and so on, in order to be able to do the task satisfactorily. (Example: Gapped Summary)

• Scoring is problematic and can be subjective

Information-transfer test

• The students’ task is to identify in the target text the required information and then to transfer it, often in some transposed form, on to a table, map, figure etc.

Cloze test

• Cloze Tests are reading passages with the blanks representing words that are deleted from the original text; the blanks are to be filled in by the reader

Types of cloze test

a. Fixed Ratio Cloze or Nth word deletion• Words are deleted systematically by counting off,

regardless of the part of speech. b. Rational Deletion Cloze• Words are deleted by part of speech or content area

vocabulary. c. Maze Technique• Three word choices are provided at each missing word

interval. d. Limited Cloze• Word choices (one per blank) are provided all together in

a word bank at the top or bottom of the page.

Tips for developing reading comprehension tests

• Make sure your assessment matches your reading program. Test the skills you have taught

• Sample a range of reading subskills with different task types. Allow 4-10 items per task type

• Choose a range of text types appropriate to your program. Consider students’ background knowledge and interests in selecting texts. Familiarity with the topic aids comprehension

• Use authentic or adapted texts whenever possible• Exploit the entire text. Questions should cover all sections of a text• Assess inferencing and critical thinking. Include questions that

require students to think beyond what they see in print