caveon webinar series - discrete option multiple choice: a revolution in testing october 26 2016

Caveon Webinar Series:

Discrete Option Multiple Choice: A Revolution in

Testing

October 26, 2016

Today’s Presenter

October 26, 2016

Dr. David Foster, CEOCaveon Test Security

3

Agenda for Today

• Overview of DOMC• Why this revolutionary technology

is so important• Barriers to using DOMC• Immediate and Long-Term Benefits• Super-DOMC

© 2016, Caveon, LLC. All rights reserved. Do not copy or distribute without permission. 4

Poll

Please rate your familiarity with the DOMC item type.

I know very little about how DOMC works.I am slightly familiar with how DOMC works.I am very familiar with how DOMC works.I know everything there is to know about how DOMC works.

5

What is DOMC?Description:

Options for the multiple choice question are presented randomly one at a time accompanied by YES and NO buttons.

Options are no longer needed when the test taker answers the question correctly or incorrectly.

6

DOMC Test Drive

The best way to understand DOMC is to experience it…

…going to trydomc.com


Foster Revolutionary Rants and Raves• It greatly improves the security of my tests.• It eliminates the unfairness found in traditional

multiple choice tests.• It reduces my test administration costs.• It reduces my test development costs.• It increases the accuracy of my tests.• It expands the types of skills that can be

measured.• It accommodates a broader range of test takers.• It motivates students to prepare better for

exams.


So, if it’s that good, why isn’t everyone using it?What are the biggest concerns?


Poll

What concerns me about DOMC?

List of 13 possible concerns on 3 polling screens. Pick the ones that seem right for you.


Addressing these Concerns


I don’t know how DOMC works.

DOMC is unfamiliar to me. I’m used to normal multiple choice.

Foster Response:DOMC seems more complicated on the surface than

traditional multiple-choice, but it’s not rocket science. There is no underlying complex theory.There are whitepapers, studies and websites that can

provide information.Practice creating and using items will remove the

mystery.


My test development software doesn’t support DOMC.I can’t create DOMC items on my development system.

Foster Response:More and more test development systems support DOMC:1. Kryterion’s Webassessor2. Excelsoft’s SARAS3. ITS’s Item Workshop4. Caveon’s Scorpion

It is not difficult to build the capability into any test development software.


My test administration software/vendor doesn’t support DOMC.

I can’t find a test administration system that supports DOMC.

Foster Response:More and more test administration systems support DOMC:1. Kryterion’s Webassessor2. Excelsoft’s SARAS3. ITS’s Internet Test Driver4. Caveon does not offer test administration,

but has an interface called SEI that supports DOMC


DOMC is too costly to implement.

I can’t afford to use DOMC.

Foster Response:It’s true that DOMC is a patented technology and it

costs a nominal fee to use that patent. The average per-test fee ends up being a few cents.

There may be some programming costs.The DOMC cost-to-benefit ratio is lower than any other

testing innovation.


My examinees won’t like DOMC.DOMC will cause a customer service issue. I don’t want to respond to a lot of complaints.

Foster Response:This is fairly accurate. Every single test taker wants to see

all of the options. They will complain and try to get you to go backward.

But research has shown that the DOMC is more intuitive and simpler cognitively.

Practice has shown that examinees quickly become familiar with DOMC and find it acceptable, even a better method.

Tutorials and practice tests are helpful in familiarizing examinees with DOMC.


Quote"DOMC was a challenge! I have always been a good test taker, in part because I understand some of the tricks that can be used to make educated guesses and, in turn, elevate my score. DOMC tosses a significant amount of those test taking tricks right out the window and kept me on my toes.”

Patrick Watts, J.D., Exam Security Administrator, CFA Institute


DOMC is too complicated.

DOMC is more complicated than what we are using today.

Foster Response: DOMC is unfamiliar and seems more complicated.DOMC items are actually easier to create, and for

examinees, they are easier to answer. The complicated, often misunderstood item is the

traditional multiple choice. For example, it introduces complicated behavior, such as comparing item choices.


DOMC takes longer to answer the questions.Because options are shown one at a time, it takes longer to answer a DOMC item, and test takers will not like the longer time limits.

Foster Response:A DOMC-based test actually takes less time!Despite options appearing one at a time, several

research studies have documented that overall testing times decrease, sometimes by as much as 40%!!!


Nobody is using DOMC.I’d feel better if other programs are using it.

Foster Response:Along with other adoptions, at least two certification programs use DOMC exclusively in their certification exams:1. OKTA Certified Professional and Certified Administrator2. Certified Exam Security Professional (Caveon)3. Three other IT certification programs currently in

development4. Medical education exams in Germany5. Preschool assessments


It is more difficult to create a DOMC item.Item writers need to learn a new set of skills to create DOMC items.

Foster Response:Actually, DOMC uses a smaller number from the same

set of skills that trained item writers have.Many test writing rules (such as keeping options similar

in length) are rendered unnecessary. It is easier to write DOMC items, and writers are already trained.

Writers DO have to learn how to score the more flexible DOMC item.


DOMC causes more security problems than it solves.The DOMC item type exposes all the item content eventually.

Foster Response:Yes, eventually, all options will be exposed.DOMC creates barriers to harvesting by reducing the

rate of item content exposure.DOMC has several other security advantages.


DOMC is not a fair way to test.

It is more fair to be able to see all the options.

Foster Response:This is simply untrue. Traditional MC shows all of the

options, giving those with better test taking skills an unfair advantage.

DOMC is able to remove sources of unfairness in standardized testing contributing to a more inclusive testing experience. It levels the playing field!


Converting to DOMC is not worth the effort.It’s a hassle to convert our existing exams, and change our examinee experience. How do we know the change will be worth it?

Foster Response: How much is peace of mind worth? How much would

you save to stop or postpone breaches? What if you could make better decisions from the test scores?

Return on Investment for DOMC is difficult to calculate as there are many intangibles. But the cost of implementing DOMC is easily returned 10-fold by the benefits.


DOMC is not psychometrically sound.DOMC does not have decades of experience behind it. How do I know it really works?

Foster Response: Research conducted so far has shown that DOMC

produces accurate and valid results.It performs better psychometrically than the time-

honored multiple-choice question.


Quote“Based on the results of these studies, there appear to be no psychometric reasons for excluding DOMC from testing programs.”

Neal Kingston, Ph.D., Director of Achievement and Assessment Institute, University of Kansas


10 Major Benefits of DOMC for High-Stakes TestingSecurity (N=3)Measurement (N=1)Fairness (N=3)Test Development (N=1)Test Administration (N=2)

These are 10 benefits out of 23. The others may be just as compelling.


Test SecurityDOMC reduces item content exposure with no loss of measurement precision.


Quote“Okta's laser beam focus on security is what led us to…adopt the DOMC format for our new certification program. We are enthusiastic about this modern innovation in item development and its ability to prevent test fraud because our priority is to protect the value of the credential for those that earn it.”

Laura Brumage, Vice President, Training at Okta


Research: Reduced Content ExposureFoster & Miller (2009) found that only about 2.5 options, on the average, were required in order to score an item as correct or incorrect.


Test SecurityDOMC prevents item harvesting and deters thieves and cheaters.

32

RESEARCH: College Students were asked:“Compared to traditional multiple choice, using DOMC, how difficult would it be to cheat on tests?” (…steal test content?)


Quote"Item and test creation is an art form that combines good science, good business, and user-friendliness, among other things. I can see why the DOMC item type is said to be the ultimate secure item since the examinee never sees the stem and all the answer options on the same screen.”

Kim Brunnert, Ph.D., Director of Education, Elsevier Education

34

Quote“DOMC testing may also considerably reduce the danger of unwanted copying and sharing of test items because many answer options do not have to be presented at all.”Sonja Willing, Martin Ostapczuk, and Jochen Musch, University of Dusseldorf, in Advances in Health Science Education, 2014


Test SecurityDOMC provides an inherent “marking” of a test, almost as reliable as a fingerprint, allowing for quick tracking an item harvester.


MeasurementDOMC removes the influence of irrelevant factors, making the test score more accurate and therefore more useful.


Quote"I think that the scoring and analysis that comes from the DOMC items will be a useful addition to Psychometrics and Measurement.”

Kim Brunnert, Ph.D., Director of Psychometrics, Elsevier Education


FairnessDOMC removes unfair advantages some test takers have. Puts everyone on an equal basis.


FairnessDOMC reduces the impact of poor reading ability.


FairnessDOMC expands the reach of testing to underserved populations (e.g., young children; second language learners; cognitively impaired individuals; people unfamiliar with testing).


Test DevelopmentDOMC reduces test development costs because they are simpler to create and have a longer “shelf-life”.


Quote: Jamie Mulkey, Ed.D, CESP Program Director“By using DOMC items for the CESP Generalist exam we can extend our item development timelines and limit item exposure.”


Test AdministrationDOMC is easier to use on mobile devices as it requires less screen “real estate”.


Test AdministrationDOMC generates significant time savings, reducing test administration costs.


Research: Time SavingsWilling, Ostpczuk, &Musch (2014) found savings of 39% in the time needed to administer medical education tests.

Foster & Miller (2009) found savings of at least 10% for college students taking psychology exams.


Learning and MotivationUnable to use test taking skills, or to obtain pre-knowledge of the test questions, examinees will be required to learn the proper knowledge or skills.


Quote"The largest difference in my mind is removing the ability to eliminate some of the answer choices. I think this could have a profound change on learning if implemented on a large scale. Could it provide a stimulus for deeper learning and understanding if students knew they would not have the opportunity to choose the correct answer but had to determine if it was in fact the correct answer? Only time will tell.”

Joseph Wolfe, Director of State and Federal Testing, Charlotte Mecklenburg Schools, North Carolina


All of these benefits lead to greater……Validity of test scores


Quote“Using DOMC items has allowed the CESP program to feel confident about candidate test results.”

Dr. Jamie Mulkey, CESP Program Director


What is Super-DOMC?ALL of the benefits are boosted even more if Super-DOMC is used.

Super-DOMC items are those where many more incorrect and correct options are produced.


Feasibility of DOMCWith research support, new tools, and operational program use, DOMC remains a solid option for testing programs

52

Questions?

Thank you!

Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”

53

http://www.caveon.com/blog