The Effectiveness of Using Multiple Choice Tests versus Short-Answer Test in Testing Grammar

Multiple choice is a form of assessment in which respondents are asked to select the best possible answer. Multiple choice format is most frequently used in educational testing. Meanwhile short answer test is the test of people’s ability to demonstrate an understanding of relationship between the course content, course objective and the practical and theoretical perspectives used to understand the course. When the students are going to have short answer test they study hard since they have to integrate what they have learned in lectures, readings, and discussions over the duration of semester. Unfortunately, the students’ score of short answer test is lower than students’ score of multiple choice test. Thus, this study was aimed to investigate the effectiveness of using multiple choice versus short answer question in testing grammar. The method that was used in this study was quantitative research where the data were analyzed using t-test. From the means of two groups (participants) were known that H1 was accepted. It means that there was difference between multiple choice test and short answer test in testing grammar. Multiple choice test more effective than short answer test in testing grammar.

Key words: Short answer, Multiple choice, Testing Grammar.

Introduction

The importance of testing in education makes it an important topic of conducting

research in testing field. Evaluation by teacher-made tests in schools is an important part of the

educational system and crucial area for research (Haynie, 1990a, 1990b, 1991, 1992; Mehrens &

Lehmann, 1987, Wiggins, 1993). According to Heaton’s study, there are many reasons for giving

a test, but however the teacher prepare a test, he or she must think of the real purpose of the test

which is given to the students.

One of the most important reasons is to find out how well the students have mastered the

language, areas and skills that have just been taught. These tests look back and they are called

Progress tests. If the teacher tests what has been recently taught and also practiced then there is

an expectation that students will score fairly high marks. If it does not happen and many students

fail, something must be wrong either with the teaching itself, the materials or students’

understanding of the syllabus. Also progress tests at the end of a unit or a semester should reflect

progress, not failure. The teacher should try to give progress tests regularly, but it is very

important to avoid over-testing, which de-motivates the students. “The best progress test is one

which students do not recognize as a test but see as simply an enjoyable and meaningful

activity”, states Heaton (9).

On the other hand, a classroom test does not focus only on students’ knowledge or

problems with the language, but furthermore, it is concerned with evaluation for the purpose of

enabling teachers to increase their own effectiveness by making adjustment in their teaching. As

a result, those classroom tests should enable groups of students or individuals in the class to

benefit more. The inquiry helps the teacher to select and evaluate the effectiveness of the

syllabus as well as the methods and materials that he or she uses. Furthermore, the test results

may indicate the whole area of the language syllabus, which have not been understood well by

the class. If, for example, eight or more students make a similar mistake in the Present Perfect

Tense, the teacher should take this problem area into account when planning further teaching.

Further, Heaton notes that another important function of testing is to encourage students.

Because most people like doing things at which they are good, the test should be made to show

students’ progress, and on the other hand, students will be usually good at the things, which they

like. It means that a well-constructed classroom test may be effectively used to motivate

students. What is more, if the details of their performance are given as soon as possible after the

test, the students should be able to learn from their weakness. Heaton concludes, that in these

ways a good test can be used as a valuable teaching device (Heaton, 1990).

Jeremy Harmer (2001) mentions two basic characteristics of a good test: Validity and

Reliability. According to Harmer’s statement, a test is valid if it tests what it is supposed to test.

Thus it is not valid, for example, if it is supposed to test writing ability with an essay question

that requires special knowledge of history or biology. In addition, to be reliable, a good test

should give consistent results. For example, if the same group of students took the same test

twice within two days, without reflecting on the first test before they wrote it again, they should

get the same results on each occasion. In practice, these principles should be kept: make the test

This paper actually will discuss more about testing grammar using two kind different

method of testing. Those two methods of testing grammar that will be discussed here are

multiple choice test and short answer test. My students say that they will study hard if they are

going to have a short answer test for their testing and usually they will be less of study if they are

going to have multiple –choice test for their testing, in this case the testing is in testing grammar.

Even though they say that they have studied hard for short answer testing but their score

in this type of testing is low. On the contrary, they are more successful at completing multiple

choice items as they may also guess and it is probably easier work for them. That is the reason

why I have decided to investigate the effectiveness of using Multiple –choice test versus Short

answer test in testing grammar.

Description

Multiple choice

Multiple choice is a form of assessment in which respondents are asked to select the best

possible answer out of choice from the list

(http://www.en.m.wikipedia.org/wiki/multiple_choice). Thus, the respondent have to choose the

answer that they think is right from several possible answer that are listed in the question paper.

Multiple-choice tests have been used extensively in recent years for assessment purposes.

Multiple choice questions were considered to be an ideal test instrument for measuring students’

knowledge of grammar and vocabulary (Harmer 2001). In addition, Heaton (1990) describes

multiple choice questions as a device that tests the ability to recognize sentences which are

grammatically correct. Wherever possible, the items of tests should be set in context. If the

teacher wants to concentrate on a certain area of grammar, he or she should put the item into a

short two-line dialogue. This is better than providing no context at all. (Zlabkova, 2007)

Multiple-choice tests have reportedly been used to good effect in courses ranging from computer

programming (eg Kuechler & Simkin, 2003) to physical education (eg Ayers, 2001).

The advantages of multiple-choice tests to be found in the literature (eg Epstein et al,

2002; Higgins & Tatham, 2003; Kuechler & Simkin, 2003) include that they can:

• test knowledge quickly within large groups

• be used to provide quick feedback

• be automatically scored

• be analysed with regard to difficulty and discrimination, and

• be stored in banks of questions and re-used as required

The literature (eg Wesolowsky, 2000; Paxton, 2000; National Center for Fair & Open

Testing, n.d.) also lists many disadvantages, including that multiple-choice tests can :

• take a lot of time to construct

• test knowledge and recall only

• never test literacy, or ability to analyse

• never test creativity, or unique thinking, and

• encourage students to take a surface approach to learning

There are four steps of constructing multiple choice items described by Heaton (1988)

Step 1: The first step is to know what the teacher wants to test and to have the sentence

testing that. The teacher may think the sentence up or use sentences and errors made by

students in their free composition and open-ended answers to questions. I like tea but I

haven’t / don’t like coffee.

Step 2: Next the teacher writes out the sentence substituting a blank for the area being

tested and then writes in the correct option and the distractor which the student has

provided. I like tea but I .......................coffee. A don’t like B haven’t like

Step 3: Now the teacher adds two more distractors. Again he or she may go to the written

work of the students to provide these distractors. But if he or she cannot find any suitable

errors without too much difficulty, the teacher uses his or her own experience and

knowledge of the target and native languages. Of course, it is necessary to be very careful

and to not give more than one correct option. There is also a possibility to change the

positions of the distractors. I like tea but I .......................coffee. A doesn’t like B don’t

like C haven’t like D like

Step 4: Finally the teacher should check to be sure that other options are not correct and

ask his or her fellow teachers to try the test to find possible imperfections.

Short answer test

One Method of testing that has received little attention in the literature, however, which is

popular in many educational settings, is the use of short answer test items (Haynie, 1983). Short

answer test is testing that test people’s ability to demonstrate an understanding of the relationship

between the course content, the course objectives and the practical and theoretical perspectives

used to understand the course (http://ar.cetl.hku.hk/am_saq.htm). Short answer question are

open-ended question that require the students to create the answer. They are commonly used in

examination to assess the basic knowledge and understanding. Short answer test does not have a

generic question. Question may have complete the sentence, supply the missing word, short

descriptive or diagrams with explanation. The answer usually short, from one to a few line.

(http://ar.cetl.hku.hk/am_saq.htm).

Example of Short answer test

1. Diana is _________ (write) a letter.

Like multiple choice test, short answer test has several advantages as well. The advantages of

short answer test are:

Short answer test are relatively fast to mark and can be marked by different assessors,

as long as the question are set in such a way that all alternative answers can be

considered by the assessors.

Short answer test are also relatively easy to set compared to many assessment

methods.

Short answer test can be used as a pert can be use for summative and formative

assessment, as the structure of short answer tests are very similar to examination

question.

Short answer test are relatively easy to prepare. (http://ar.cetl.hku.hk/am_saq.htm)

It is not an easy thing to make short answer test. People need to follow the step in constructing

the short answer test. These are several steps to construct the short answer test:

1. Design short answer items that are appropriate assessment of the learning objective.

2. Make sure that the content of the short answer question measures knowledge appropriate

to the desired learning goal.

3. Express the question with clear wordings and language which are appropriate to the

student population.

4. Ensure that the item clearly specifies how the question should be answered (e.g. Students

should answer it briefly and concisely using a single word or short phrase?

5. Write the instruction clearly so as to specify the desired knowledge and specify response.

(http://ar.cetl.hku.hk/am_saq.htm)

Why Testing Grammar

Grammar is the structural foundation of our ability to express ourselves. Grammar

is important because it is the language that makes it possible for us to talk about language.

Grammar names the types of words and word groups that make up sentences.

The testing grammar is one of the mainstays of language testing. While such tests

test the ability to either recognize or produce correct grammar and usage, people do not tests

the ability to use the language to express the meaning. However, it can be argued that a basic

knowledge of grammar underlies the ability to use language to express meaning, and so

grammar test do have an important part to play in language programs.

People associate grammar with errors and correctness. Grammar can be part of

literature discussion, when teacher and the students read the poetry or stories, thus, knowing

about grammar means finding the out that all languages and all dialects follow grammatical

patterns. (http://www.grammar.about.com/od/basicsentencegrammar/a/grammarintro.htm)

From the statement above, can be concluded that grammar is very important in

Language where in this case is English, thus testing grammar is chosen in this research to be

analyzed in case of comparing the effectiveness of using multiple choice test and short

answer test in testing grammar.

Methodology

Population and Sample

There were 64 students tested. All of them attended the XI grade at Senior High

School, they were from the same school, the students at grade XI chosen since they have got

more materials about grammar. Class XII actually had more knowledge about grammar but

they were deleted from this population since in this grade the students had to focus on

National examination. The 64 students divided into 2 groups: (a) Multiple-choice test

(Group A, n= 32), (b) Short answer test (Group B, n=32). Those 64 students has the same

level of intellectual, it can be seen from the result of the mid test which had been conducted

before. The scores of those 64 students are quite similar or did not reveal the significant

difference.

Design

The design of this research was a quasi-experimental, a type of research design which

includes experimental and control groups without random sampling (Hatch and Lazarraton,

1991). Those two groups of students were taught by the same materials whether it was a

Group A or a Group B got the same steps in teaching materials. The steps began from the

easiest one for example the material of to be, and then move to the other steps until they

reached tenses steps. After knowing all the materials then the test was prepared. The test

was well-arranged and, of course, clear instructions and the example were given at the

beginning. For the time testing, each group has equal time in doing the test.

Instrument and Data Collection

The score of the students’ tests were used to know the effectiveness of using the

multiple choice test versus short answer test in testing grammar. They were collected

through thirty multiple choice items and twenty short answer question. The test was divided

into three parts: the first part was focused on Present Simple tense, the second on Present

Continuous and the third one concern both the tenses mixed together.

Data Analysis

The results of this study were based on the tests which were conducted. Quantitative

analysis was used to see the effectiveness of using multiple choice versus short answer test in

testing grammar. Score of the students were analyzed through t-test since this test used to

determine whether there was a different significant between two means of the group then was

going to be known whether H1 or Ho which was rejected. If H1 accepted it means that multiple

choice more effective than short answer in testing grammar, however, if Ho accepted then short

answer more effective than multiple choice in testing grammar.

Finding and Discussion

This section presents the research findings and the discussion based on the analysis of the

data collected from the use of multiple choice items versus short answer test in testing grammar.

The research findings and the discussion are presented on the basis of the result of the tests. The

procedure, compared the means of the Group A and Group B to find out their significant

difference. If any, there might be a possibility that the means score of the two groups are

different. The significant difference can be seen after the t-value has been obtained. The formula

is:

t = Difference between two sample means Standard error of differences between means

t = _______________ M1-M2____________________________

∑ x 12−¿¿ + ∑ x 22−¿ 1_ + 1_ (N1-1) + (N2-1) N1 N2

Explanation: M1 : Mean score of group AM2 : Mean score of Group B¿ : Square the total score group A

¿ : Square the total score group B∑ x 12 : Sum the square in Group A∑ x 22 : Sum the square in Group BN1 : The number of participants of group A

N2 : The number of participants of group B

The first step to get the data to be analyzed was conducting the test. In conducting the test

both two group are given the test with different type of test, Group A with multiple choice test

and Group B with short answer test. If the scores have been got, then continued by analyzing the

data could be started. From the test which had been conducted, the scores which are got by the

students are presented in the following table:

Table of students’ scores NO Group A ( Multiple

Choice test)Squares Group B ( Short

answer tets)Squares

1 78 6084 66 43562 77 5929 71 50413 81 6561 47 22094 73 5329 57 32495 71 5041 64 40966 78 6084 51 26017 74 5476 74 54768 72 5184 77 59299 71 5041 59 348110 69 4761 53 280911 76 5776 49 240112 74 5476 65 422513 72 5184 66 435614 71 5041 51 260115 77 5929 59 348116 77 5929 48 230417 76 5776 66 435618 69 4761 65 422519 78 6084 68 462420 77 5929 49 240121 79 6241 71 532922 82 6724 73 230423 86 7396 61 372124 88 7744 68 462425 75 5625 73 532926 73 5329 69 4761

27 80 6400 67 448928 75 5625 70 490029 72 5184 74 547630 85 7225 73 532931 82 6724 78 608432 71 5041 76 5776

M1 : 76.26M2 : 64.41¿ : 5948721

¿ : 4235364∑ x 12 : 186633∑ x 22 : 135080N1 : 32N2 : 32

The normal distribution scores are needed to fulfill the requirement, in this case, t-test

(Hatch and Lazaraton, 1991: 164). Since the sample of the study was more than 30 (32 students

in experimental and 32 students in control group) the normality test was not conducted. Hatch

and Farhady (1982: 64) state that if the N (number of cases) is 30 or more, the distribution of

scores of that random sample is close enough to a normal distribution.

The score of the students which were as the data had been got, it means the next steps

could be started. The next step was analyzing the data, counted the several numbers which were

needed to get the t-value of the data. Here, t-test was used to test or compare two means of both

of the groups. The significance criteria were gained by calculating t-value with the assumption

that Ho is accepted if t-value < t-table while Ho is rejected if t-value > t- table.

From the begin calculating process which had been done, could be explained that the 32

students of Group A were got the result of mode, mean, median and SD were like these; mode

of this group was 77, the median was 76, mean of this group was 76.26 and the SD of this group

was 4,65. Meanwhile for 32 students of Group B were got the result of mode, mean ,median and

SD were like these; for mode of Group B was 66, median of 32 students of Group B was 66 as

well, for mean of Group B was 64.41 and SD of the short answer test group was 8,96.

For getting the t-value, it needed to calculate the square total and sum square of each

group; Group A and Group B. The sum square Group A was 186633, meanwhile the sum square

Group B was135080. After getting the sum square of each groups then the result of square total

were going to be known. For Group A the result of the square total was 5948721 then for Group

B the result of the square total was 4235364.

The calculation results of independent t-test revealed that the t-value was 6.53, while t-

table with alpha .05 and df 60 was 2.00. Thus, t-value was greater than t-table which indicated

that there was a significant difference between the Group A or Multiple choice group and Group

B or Short answer test group. Thus, Ho was rejected and H1 was accepted. It can be concluded

that testing grammar using multiple choice more effective than using short answer .

CONCLUSIONS

This last section presents the conclusions drawn from the research findings and

discussion in the previous chapter. This research aimed to discover the effectiveness of using

multiple choice items versus short answer test in testing grammar. The data was collected from

tests. Both of groups were conducted the test at the same times but different types of the test. The

first group or Group A used multiple choice test items and the second group used the short

answer test. Those both groups have the same level of intellectual, so that, the validity of the test

result can be guaranteed. Regarding to aimed or purposed of this research, the data gained from

the independent t-test, it showed that there was a significant difference between Group A and

Group B, who were taught by us of the scores result. It can be concluded that testing grammar

using multiple choice more effective than using short answer test.

REFFERENCES

Ayers. S.F. (2001). Developing quality multiple-choice tests for physical education Journal of Physical

choice questioning Assessment and Evaluation in Higher Education. Bath: Jun 2000. Vol. 25, Iss. 2;

p. 109 Education, Recreation & Dance. Reston: Aug 2001. Vol. 72, Iss. 6; p. 23

Epstein, M. L., Lazarus, A. D., Calvano, T. B., Matthews, K. A. et al. (2002). Immediate feedback

assessment technique promotes learning and corrects inaccurate first responses

Harmer, Jeremy.( 2001.) The Practice of English Language Teaching. Harlow: Pearson Education.

Hatch and Farhady. 1982. Research Design and Statistic for Applied Linguistics. Massachusetts:

Newbury House Publisher, Inc.

Hatch and Lazaraton. 1994. The Research Manual; Design and Statistic for Applied Linguistics.

Massachusetts: Heinle & Heinle Publishers.

Haynie, W, J. (1992). Post hoc analysis of test items written by technology education teaqchers. Journal

of Technology Education, 4(1), 27-40.

Haynie, W.J. (1990a). Effects of tests and anticipation of tests on learning via videotaped materials.

Journal of Industrial Teacher Education, 27 (4), 18-30.

Haynie, W.J. (1990b). Anticipation of tests and open space laboratories as learning variables in

technology education. Proceedings of the 1990 North Carolina Council on Technology Teacher

Education Winter Conference. Camp Caraway, NC: NCCTTE.

Haynie, W.J. (1991). Effects of take-home and in-class tests on delayed retention learning acquired via

individualized, self-paced instructional texts. Journal of Industrial Teacher Education, 28(4),

52-63.

Haynie,W. J. (1983). Student evaluation: The teacher’s most difficult job. Monograph Series of the

Virginia Industrial Arts Teacher Education Council, Monograph Number 11.

Heaton, J. B.( 1988) Writing English Language Tests. Harlow: Longman.

Heaton, J. B.( 1990) Classroom Testing. Harlow: Longman.

http://ar.cetl.hku.hk/am_saq.htm. Accessed On December, 4th 2013

http://www.en.m.wikipedia.org/wiki/multiple_choice. Accessed On December, 4th 2013

http://www.grammar.about.com/od/basicsentencegrammar/a/grammarintro.htm. Accessed On

December, 4th 2013

Illinois State University (2001). Centre for theAdvancement of Teaching. Journal of Applied Statistics.

Sheffield: Sep 2000. Vol. 27, Iss. 7; p. 909

Kuechler, W. L. & Simkin. M. G. (2003). How Well Do Multiple Choice Tests Evaluate Student

Understanding in Computer Programming Classes? Journal of Information Systems Education.

West Lafayette: Winter 2003. Vol. 14, Iss. 4; p. 389

Paxton. M. (2000). A linguistic perspective on multiple

choice questioning Assessment and Evaluation in Higher Education. Bath: Jun 2000. Vol. 25, Iss.

2; p. 109

Wesolowsky. G. O. (2000). Detecting excessive similarity in answers on multiple choice exams.

Zlbkova, L. 2007. Testing Grammar: Using Multiple Choice Tests versus Translation. Masaryk

University Brno.

The Effectiveness of Using Multiple Choice Tests versus Short-Answer Test in Testing Grammar

(Research Based Paper of Language Testing)

ARRANGED BY

TIARA GARDENIA RESMITA

1202213

School of post GraduateEnglish Education Study ProgramIndonesia University of Education

2013