challenging student intuition with item design€¦ · institute of chemistry education challenging...

Institute ofChemistry Education

Challenging student intuition with item designMICER III Zoomposium, 18.06.2020, Prof. Nicole Graulich

#nik_kola82 Graulich CER Group: www.uni-giessen.de/dc

Graulich CER Group: www.uni-giessen.de/dc


How do people make decisions and what does this mean for item design in CER?



PRE-READING:Graulich, N., Hedtrich, S. and Harzenetter, R. Explicit

versus implicit similarity – exploring relational conceptual understanding in organic chemistry, Chem.

Educ. Res. Pract., 2019, 20, 924-936.

René Harzenetter Sebastian Hedtrich

Roadmap

1. Intuitive reasoning impacts students’ decision-making processes

2. An exploratory study to challenge student surface reliance – approaches and struggles (pre-reading)

3. Where to go from there? - Ideas for continuing research



How do people make decisions?

System II Thinking• analytical• slow• not effortlessly

System I Thinking• fast• intuitive• effortlessly• unconcious

Evans, J. S. B., In two minds: dual-process accounts of reasoning. 2003 Trends in Cognitive Sciences, 7(10), 454-459.



Which of these two cities is larger?

Essen Frankfurt

A B

Goldstein, D. G., & Gigerenzer, G. 1999. The recognition heuristic: how ignorance makes us smart. InG. Gigerenzer, & P. Todd (Eds.), Simple heuristics that make us smart. New York: Oxford University Press.

Knowing less can, depending on the question, be more effective than knowing more.



Which person would you consider more friendly?

A B

Monin, B., The Warm Glow Heuristic: When Liking Leads to Familiarity, 2003, Journal of Personality and Social Psychology, 85, 1035-1048.

We tend to substitute a difficult (or even impossible) question by a simpler one.

attribute substitution effect



Which are the appropriate reagents for this reaction?Attribute substitution effect

Graulich N., Intuitive Judgments Govern Students’ Answering Patterns in Multiple-Choice Exercises in Organic Chemistry, 2015, J. Chem. Educ., 92, 205-211.

Hanna: “I would go with d.) There are two chlorines in the product, so with Cl2 and CH2Cl2, I guess, you have enough chlorines to make it.“



„Students often apply rules memorized from particular examples to new questions which they deem to be similar based upon surface features of the molecules. […],

which lead them to incorrect answers.“

Cl+ CH3OH methanol

+ NaI acetoneBr

de Arellano, D. C. R. and Towns, M., Students understanding of alkyl halide reactions in undergraduateorganic chemistry, 2014, Chem. Educ. Res. Pract. 15, 501-515.



„Students are engaged in rote memorization of what features are related to nucleophilic and electrophilic behavior, rather than try to more deeply comprehend the

relationships between those features and functionality.“

Anzovino, M. E. and Bretz, S. L. Organic chemistry students' ideas about nucleophiles andelectrophiles: the role of charges and mechanisms, 2015, Chem. Educ. Res. Pract., 16, 797–810.



„The student focused most of the attention on those explicit structural cues that differentiated the three chemical compounds, implicitly discarding (i.e., benzene ring) or explicitly eliminating (i.e., –OH group) features that were actually relevant for the

successful prediction of acid strength.”

McClary, L.; Talanquer, V., Heuristic Reasoning in Chemistry: Making decisions about acid strength. Int. J. Sci. Educ. 2011, 33 (10), 1433-1454.



Acid-Base theory

Stereochemistry

Nucleophiles-Electrophiles

Polarity

Stability

Driving force

Kinetics

Electron movement

Reaction types

Electronegativity

Steric hindrance

Thermodynamics

π- and σ-bonds

Ring strain

Geometry

Resonance

Hydrogen bonds

Hyperconjugation

Review: Graulich, 2015, Chem. Educ.

Res. Pract., 16, 9-21.

students’ success in OC depends on the ability to:

(1) establish connections between the chemical properties and the structural representation

(2) consider implicit properties of molecules(3) determine which implicit property is relevant

in a problem context, despite explicit cues.



Acid-Base theory

Stereochemistry

Nucleophiles-Electrophiles

Polarity

Stability

Driving force

Kinetics

Electron movement

Reaction types

Electronegativity

Steric hindrance

Thermodynamics

π- and σ-bonds

Ring strain

Geometry

Resonance

Hydrogen bonds

Hyperconjugation

What type of items would capture this relational conceptual

understanding?

Review: Graulich, 2015, Chem. Educ.

Res. Pract., 16, 9-21.



A B C

Item Design Background

Inspired by:

• Variation theory of learning (Bussey, T. J. et al. 2012)

• Learning through contrasts (Schwartz, D. L. et al. 2011)

Explicit versus implicit similarity



Which two food items contain more sugar?

A B C

Item Design Background

Inspired by:

• Variation theory of learning (Bussey, T. J. et al. 2012)

• Learning through contrasts (Schwartz, D. L. et al. 2011)

Explicit versus implicit similarity

Poll #2

A: A and B

B: B and C

C: C and A



Which two nucleophiles would react similar with this bromalkane in a substitution reaction?

Item Design

A B C

Br ?

O

O

O

C N

A

B

C

Br ?

O

O

O

C N

A

B

C

Br ?

O

O

O

C N

A

B

C


Institute ofChemistry EducationItem Design (SN1/SN2 reactions)

General Prompt: Indicate the two structures that would react similar in a substitution reaction.

nucleophilicity

Br ?

O

O

O

C N

A

B

C

Br BrBr

CBA

hyperconjugative effects

Cl

C N?

SN2

OH

A B C

CCl4

Br OCH3Cl

A B C

solvent effects

leaving group



LEADING QUESTION

How do students judge a comparable reactivity in items that are explicitly distracting?

How does students’ level of elaboration, when asked to provide a reason for their choice, relate to their performance?

Quantitative study, 156 students (chemistry majors / student teachers), 1st year OC

Paper-Pencil:• Part 1 – Multiple Choice• Part 2 – Multiple Choice and elaboration• Instrument: substitution reactions, 68 items in total• Concepts: Leaving group, hyperconjugative effects, solvent effects, nucleophilicity

Analysis• statistical analysis• rubric for the coding student elaboration

Study Design


Institute ofChemistry EducationHow to categorize students' elaboration?

A Both are secondary halogen alkanesB Both react similar as both are good leaving groupsC Halogens are able to better stabilize a negative charge when leaving the moleculeD A and B have a high electronegativity.

Indicate the two structures that would react similar in a substitution reaction.

Br OCH3Cl

A B C

leaving group

Students’ answers for choosing A and C:


Institute ofChemistry EducationRubric for students’ elaboration

Student Examples:explicit implicit

descriptive

functional

Both are secondary halogen alkanes

Both react similar as both are good leaving groups

Both halogens are able to better stabilize a negative charge when leaving the molecule

A and B have a high electronegativity

E1

E2

E3

E4

Deductive coding

a. Both are secondary halogen alkanesb. Both react similar as both are good leaving

groupsc. Halogens are able to better stabilize a

negative charge when leaving the moleculed. A and B have a high electronegativity.


Institute ofChemistry EducationResults

This is what we got from a first analysis!

These items were not all answered in the same way

dist

ribut

ion

of st

uden

ts’ m

ean

scor

es

item type 1

item type 2



Br OCH3Cl

A B C

leaving group

Item type 1 - supporting Item type 2 - distractingExplicit features support item solution Explicit features do not support item solution

nucleophilicity

Br ?

O

O

O

C N

A

B

C

Reorganization of items



Box Plots after reorganization

• large effect size for the item types (Cohen‘s d=1.4)

• small effect size for the elaboration prompt in both item types (type 2: d=0.31, p=0001 and type 1: d=0.26, p= 0.009)

Part 1without elaboration

Part 2with elaboration

type 1supporting items

type 2distracting items

dist

ribut

ion

of st

uden

ts’ m

ean

scor

es


Institute ofChemistry EducationStudents’ elaboration

• Students, who elaborate on the lower levels have a higher chance of solving the item of type 2 incorrectly.


Institute ofChemistry EducationWhere to go from here?

• Assessing students’ understanding can be misleading, when the correct solution is possible based on surface features.

• Combining assessment with students’ deeper reasoning, e.g. two-tiers MCQs (Treagust, D. F. 1988).

• Supporting students to activate deep conceptual reasoning does not come deliberately and already starts during instruction

What are you doing while teaching to support students to become more reflective about their intuitive approaches?

THINK



Get to know each other and exchange your experiences.

THINK What are you doing while teaching to support students to become more reflective about their intuitive approaches?

BREAKOUT

CHAT Decide which idea/question you want to share as a group via the chat box.


Institute ofChemistry EducationMaking the unconscious conscious

Students’ 1st attempt to a prompt

Analyzing worked-example, expert- or peer-solutions (e.g. work by E. Yuriev; Finkenstaedt-Quinn et al., 2019)

PROMPT Revision of 1st

attempt

Prompt to reflect on possible erroneous solutions (e.g. Talanquer, 2017)

Learning with material (e.g. conflicting animations, Kelly et al., 2017)

….

metacognitive „loop“

• How can I determine changes, what can be taken as evidences?

• Do changes last? How I can observe this?• How are learned abilities used in other

contexts?


Institute ofChemistry EducationHuge Thanks to !

... Michael & Aishling for just being fabulous

... an open and respectful CER community

... every one of you for being here today!

... a fantastic CER research group in Gießen

Reach out to us!!


Institute ofChemistry EducationResources

• Evans, J. S. B., In two minds: dual-process accounts of reasoning. Trends in Cognitive Sciences 2003, 7(10), 454-459.• Talanquer, V., Concept Inventories: Predicting the Wrong Answer May Boost Performance. J. Chem. Educ. 2017, 94 (12), 1805-1810.• Posner, G. J.; Strike, K. A.; Hewson, P. W.; Gertzog, W. A., Accomodation of a scientific conception: Towards a theory of conceptual change. Sci. Educ. 1982, 66, 211-227.• Goldstein, D. G., & Gigerenzer, G. 1999. The recognition heuristic: how ignorance makes us smart. In G. Gigerenzer, & P. Todd (Eds.), Simple heuristics that make us smart. New York:

Oxford University Press.• Monin, B., The Warm Glow Heuristic: When Liking Leads to Familiarity, Journal of Personality and Social Psychology 2003, 85, 1035-1048.• De Arellano, D. C. R.; Towns, M., Students understanding of alkyl halide reactions in undergraduate organic chemistry. Chem. Educ. Res. Pract. 2014, 15, 501-515.• Anzovino, M. E. and Bretz, S. L., Organic chemistry students' ideas about nucleophiles and electrophiles: the role of charges and mechanisms, Chem. Educ. Res. Pract. 2015 16, 797–810.• Graulich, N., Hedtrich, S. and Harzenetter, R., Explicit versus implicit similarity – exploring relational conceptual understanding in organic chemistry, Chem. Educ. Res. Pract. 2019, 20, 924-

936.• Graulich, N.; Hopf, H.; Schreiner, P. R., Heuristic thinking makes a chemist smart. Chem. Soc. Rev. 2010, 39 (5), 1503-12.• Graulich, N. The tip of the iceberg in organic chemistry classes: how do students deal with the invisible? Chem. Educ. Res. Pract. 2015, 16, 9-21.• Graulich, N. Intuitive Judgments Govern Students‘ Answering Patterns in Multiple-Choice Exercises in Organic Chemistry. J. Chem. Educ. 2015, 92, 205-211.• Graulich, N.; Bhattacharyya, G. Investigating Students’ Similarity Judgments in Organic Chemistry, Chem. Educ. Res. Pract., 2017, 18, 774-784.• Bussey, T. J.; Orgill, M.; Crippen, K. J., Variation theory: A theory of learning and a useful theoretical framework for chemical education research. Chem. Educ. Res. Pract. 2013, 14 (1), 9-22.• Schwartz, D. L.; Chase, C. C.; Oppezzo, M. A.; Chin, D. B., Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. J. Educ. Psychol. 2011, 103

(4), 759.• Kelly, R. M.; Akaygun, S.; Hansen, S. J. R.; Villalta-Cerdas, A., The effect that comparing molecular animations of varying accuracy has on students' submicroscopic explanations. Chem.

Educ. Res. Pract. 2017, 18 (4), 582-600.• Finkenstaedt-Quinn, S. A.; Snyder-White, E. P.; Connor, M. C.; Gere, A. R.; Shultz, G. V., Characterizing Peer Review Comments and Revision from a Writing-to-Learn Assignment Focused

on Lewis Structures. J. Chem. Ed. 2019, 96 (2), 227-237.• Treagust, D. F. Development and use of diagnostic tests to evaluate students’ misconceptions in science. Int. J. Sci. Educ. 1988, 10, 159–170.

Online resources:• Intuitive concept inventory: https://www.surveygizmo.com/s3/4364595/ICI• V. Talanquers Webinar: https://rsccerg.wordpress.com/2018/04/30/vicente-talanquer-webinar-23rd-may/• Stacey Bretz Webinar: https://www.youtube.com/watch?v=WfSh4r3wtR0

https://www.surveygizmo.com/s3/4364595/ICI

https://rsccerg.wordpress.com/2018/04/30/vicente-talanquer-webinar-23rd-may/

https://www.youtube.com/watch?v=WfSh4r3wtR0

challenging student intuition with item design€¦ · institute of chemistry education challenging...

Documents