Beyond MCQ: Assessment That

Increases Reading Ability

Jay Bidal, Senior Instructor

English Language Centre, University of Macau

Basic Academic Reading

• comprehending

--main ideas

--key details/support

--inferences

• note-taking--for later review (e.g. before an exam)

• finding information--scanning, skimming

• mining

“the process of reading for the purpose of culling information from a text for a specific goal” (Plakans, 2009, p. 3)

• critical reading (evaluating truth statements, arguments…)

Assessing Reading

• multiple choice (incl. True/False/Not Given)

• matching

• ordering

• information transfer/gap fill

• cloze (summary)

• short answer

• summary writing

• communicative tasks/authentic tasks

(Alderson, 2005)

Reading and Purpose

“…the purpose for which readers on tests are reading, and possibly the manner in which they are reading, may not correspond to the way they normally read such texts” (Alderson, 2005, p. 249).

“asking test-takers to respond to text passages with multiple-choice questions induces response processes that are strikingly different from those that respondents would draw on when reading in non-testing contexts” (Rupp, Ferne, & Choi, 2006, p. 441).

“While the stated objectives of academic English programs usually include preparing students for the tasks associated with successful performance at a university, the methods for assessing the degree of success students achieve in reaching these goals are widely recognized as lagging far behind” (Robinson & Ross, 1996).

Performance-based/Task-based Reading Assessment

“Task-based tests require candidates to perform an activity which simulates performances they will have to engage in outside the test situation…” (Robinson & Ross, 1996)

“Task-based performance assessments in teaching programmes have proven particularly valuable because task-based assessments can be linked to teaching outcomes, provided outcomes are defined in terms of task fulfilment, rather than purely in terms of language ability” (Wigglesworth, 2008, p. 114).

Integrated Task Assessments

“The conventional practice…of separating the skills runs counter to the common-sense observation that actual academic language use tasks routinely involve combinations of skills” (Read, 2015, p. 186).

“For academic purposes, the construct of writing ability that needs to be assessed includes the ability not only to compose coherent written texts but also to produce texts that are…related meaningfully and appropriately to ideas and information in academic sources” (Cumming, 2013, p. 3)

“…integrated language assessment…typically requires the test takers to perform a writing or speaking task after first reading and/or listening to at least two input texts that provide the necessary content for the task” (Read, 2015, p. 187).

Integrated Task Assessments: Some Issues

• task dependency

“Without adequate comprehension of source materials, learners cannot write effectively about them, thereby compromising the validity of measurements of writing abilities for learners without adequate comprehension abilities” (Cumming, 2013, p. 4).

• threshold levels of proficiency

“…without sufficient comprehension of source materials second-language writers simply cannot write effectively about them” (Ibid, p. 5).

• rater reliability

“the problem of generating unique rubrics for impressionistic scoring of unique task types, each with unique content, and of establishing interrater reliability for novel tasks” (Ibid, p. 5)

Semi-integrated Reading Task

Course Learning Outcome:

“Students will be able to identify relevant/useful information in adapted and authentic reading texts according to a given purpose.”

Task Design

• one essay writing prompt

• two adapted reading texts related by topic to the writing prompt

• students highlight/underline information in texts relevant to the writing prompt and annotate said information

• (students use information to partially complete an essay outline)

Instructional Approach

•3 cycles (different topics, Q’s, texts)•cognitive modeling (increasing levels of S input)•peer comparisons and discussions•self-evaluation (comparing with model)• feedback (in-class, office hours)

Grading

Rubric (4 criteria)

CriteriaExcellent

work

Good

work

Satisfactory

work

Weak

work Points

Relevancy

S highlights information relevant to the given

essay question.5

Content

S highlights all key information related to given

essay question.10

Annotation

S annotates highlighted information in a clear

and useful manner. 10

Manner

S does not highlight too much

or too little text.5

27/30 90%

Comments:

Instructor Exemplar

• model for comparison purposes

• open to deviations (if justified through annotations)

Semi-integrated Reading Task

Results

Low Intermediate-Intermediate Intermediate-High Intermediate

Section 1 Avg. Section 2 Avg. Section 3 Avg. Section 4 Avg.

72% 73% 78% 77.5%

Semi-integrated Reading Task: Example A

CriteriaExcellent

work

Good

work

Satisfactory

work

Weak

work Points

Relevancy

S highlights information relevant to the given

essay question.

5

5

Content

S highlights all key information related to given

essay question.

8

10

Annotation

S annotates highlighted information in a clear

and useful manner.

9

10

Manner

S does not highlight too much

or too little text.

55

27/30 90%

Comments: All highlighted information is

relevant to essay question. Some key

information is not highlighted. Annotations are

clear but wordy. Manner of highlighting hits

right balance.

Student A Text A

Semi-integrated Reading Task: Example B

CriteriaExcellent

work

Good

work

Satisfactory

work

Weak

work Points

Relevancy

S highlights information relevant to the given

essay question.

4

5

Content

S highlights all key information related to given

essay question.

4

10

Annotation

S annotates highlighted information in a clear

and useful manner.

5

10

Manner

S does not highlight too much

or too little text.

45

17/30 57%

Comments: Some irrelevant information

highlighted. Much key information is

not highlighted (esp. main idea missed).

Annotation is insufficient and not

always useful. Manner of highlighting is

generally good but better connections

could be made between highlighted

parts.

Student B Text A

Semi-integrated Reading Task: Example C

CriteriaExcellent

work

Good

work

Satisfactory

work

Weak

work Points

Relevancy

S highlights information relevant to the given

essay question.

4

5

Content

S highlights all key information related to given

essay question.

8

10

Annotation

S annotates highlighted information in a clear

and useful manner.

8

10

Manner

S does not highlight too much

or too little text.

4

5

24/30 80%

Comments: Some irrelevant information

highlighted. Some key information

missed. Annotation could relate better

to question at times. Manner of

highlighting is generally good but better

connections could be made between

highlighted parts.

Student C Text A

Student Reflections

(N=90)

“Do you think that the combined approach of highlighting reading texts, note-taking from lectures, and putting main points and key details in outlines is useful for you?”

Positive=75 Mixed=11 Negative=4

83.3% 12.2% 4.5%

Student Reflections: Positive

“Yes, it is very useful. Because when I re-read my article, I don’t need to read all of it again. Also, the highlighting can make me easily to understand the content of this article.”

“Yes, I think it is useful for me. It is because it helps me to have a clear understanding on the topic and I can be trained to find out the useful sources from the article. Before this semester, I always miss some useful information and highlight some texts that are unrelated to my essay.”

“I think it is a bit hard but really useful for me. Not only in English, but also useful for my Chinese class. I can highlight the main points when I read texts.”

Student Reflections: Negative“When I did the test, I didn’t understand clearly what should I highlight. The important points are always far from what you teach in class. To be honest, I think the right highlighting parts seem to have no connection with topic or questions. Maybe we think in different ways, or this is the gap between different cultures.”

“For the highlighting reading tests, I think you can’t give the mark for it. Because everybody has their own opinion, maybe he think it is important and you may not. It is difficult to give the mark because there is not a normal standard.”

“The purpose of the combined approach is great, but I will doubt if they really work…we don’t really have time to practice all of them… Especially the skill for highlighting. I still don’t get the point of doing it but knowing that I am not good at it.”

“It’s useful, but not really an interesting topic. It is very useful skills that I can apply immediately on research I was doing on other subjects, but I think it can be presented in a more interesting way.”

Assessing Academic Reading

Conclusion

• integrated tasks are worth exploring for teaching and assessing academic skills

• a semi-integrated highlighting/annotating task is an interesting approach to assessing students’ academic reading ability separately from productive skill ability (diagnostic purposes?)

• students tend to appreciate it as a learning point, with caveats

• need to investigate more how valid/reliable the approach is (across instructors/institutions/countries)

Academic Reading: Purpose

•communicate to students why they are to do a reading•set specific tasks•pose key questions• link reading to higher thinking skills•consider discipline-specific reading needs, resources, approaches (model?)

Critique and Discussion

How does this reading assessment tool fare in light of Prof. Carless’ 8 principles of assessment?

References

Alderson, J. C. (2005). Assessing reading. Cambridge: Cambridge University Press.

Cumming, A. (2013). Assessing integrated writing tasks for academic purposes: Promises and perils. Language Assessment Quarterly, 10(1), 1-8.

Plakans, L. (2009). The role of reading strategies in integrated L2 writing tasks. Journal of English for Academic Purposes, 8(4), 252-266.

Read, J. (2015). Assessing English proficiency for university study. UK: Palgrave Macmillan.

Robinson, P., & Ross, S. (1996). The development of task-based assessment in English for academic purposes programs. Applied Linguistics, 17(4), 455-476.

Rupp, A. A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language testing, 23(4), 441-474.

Wigglesworth, G. (2008). Task and performance based assessment. In Encyclopedia of language and education (pp. 2251-2262). U.S.: Springer.

Contact Info

Jay Bidal

Senior Instructor,

English Language Centre,

Faculty of Arts and Humanities,

University of Macau

jbidal@umac.mo