Quantitative SOTL Research

MethodsKrista Trinder, College of Medicine

Brad Wuetherick, GMCTEOctober 28, 2010

Session outcomesO Identify components of good survey

questionsO Choose an appropriate research

methodO Willingly and enthusiastically

participate in more workshops in thisO Meet and work with other dedicated

and scholarly teachers

Quantitative ToolsO StatisticsO NumbersO Measurement

O Surveys/Questionnaires are the most common tool.

Survey SelectionO Good to use pre-existing instruments

that have been found to be reliable and valid.

O May have to create your own if no suitable tools exist.

Creating a SurveyO Decide what you want to measureO Generate a large number of itemsO Decide on question formatO Have experts review your itemsO Consider including validation itemsO Give items to a development sample

(some suggest 300)O Evaluate itemsO Adjust scale length

Examples of Survey Questions

O This course was enjoyable and provided me with information that will benefit my future career.O A. Strongly DisagreeO B. DisagreeO C. NeutralO D. AgreeO E. Strongly Agree

Examples of Survey Questions

O I dislike courses that do not include small group work.

O Better: I like courses that include small group work.

Good Survey QuestionsO Keep items short and concise.O Aim for a Grade 6 reading levelO Avoid double negativesO Avoid double-barreled items (only

ask one thing per question)O Avoid ambiguous pronoun references

and phrases with unintended meanings

Quantitative Approaches

O Experimental

O Comparative and Associational

O Descriptive

ExperimentalO Randomized Experimental Approach

O CausalityO Gold StandardO Randomly assign participants to

experimental and control groups

Quasi-ExperimentalO Similar to an experimental approach,

but without random assignment.

O Poor designs: O One group posttest onlyO One group pretest-posttestO Posttest only with non-equivalent

groupsO Better design:

O Pretest-posttest non-equivalent comparison group

Comparative and Associational ApproachesO Examine effect of an attribute

independent variable on the dependent variable.

O No random assignmentO No control over independent variables

(IVs)

O Comparative: ordered IVs with few levels (men, women).

O Associational: continuous IVs with more levels (age, grades)

Descriptive ApproachO Purpose is to summarize data

O One variable; no comparisons or relationships investigated

O Report percentages, means, frequencies etc.

Discussion of Approaches

O What approach might you use?

Parametric vs. Non-Parametric

O Usually use parametric analyses, which compare means (i.e., ANOVAs, t-tests).

O Sometimes it is more appropriate to conduct non-parametric analyses, which compare distributions (i.e., Chi-square). O Very small sampleO Different data collection methods

ReliabilityO Shows similar performance at

different times.O Sources of reliability:

O Test-retestO Internal Consistency (i.e., Cronbach’s

alpha – should be at least .70)O Rater Consistency (i.e., inter-rater

reliability)

Content ValidityO Extent to which the content of the

instrument measures what you want it to.

O Define your concept.O See how it is represented in the literatureO Develop test items assessing this

concept.O Expert judges review items.O See handout

Criterion ValidityO How well your instrument compares

to an outside criterion that is known to measure the same concept.

O E.g., How well does self-reported voting correlate with actual voting shown through records?

Construct ValidityO The extent to which a measure

correlates with other measures that should be related.

O Expect correlations with measures that measure the same thing (convergent validity).

O Expect low correlations with measures that should not be related (discriminant validity)

Reliability and ValidityO A measure can be reliable without

being valid, but cannot be valid without being reliable.

O See cartoon

Mixed MethodsO Do not have to use strictly

quantitative or qualitative tools!

O Research projects often benefit from including both quantitative and qualitative methods.

Choosing your methodO See handoutsO Work independentlyO Exchange thoughts and ideas: consultO More individual workO Leave at your own discretion

Thank you!See you on November 25th!