week 3 etec 668 quantitative research in educational technology dr. seungoh paek january 29, 2014
TRANSCRIPT
Week 3 ETEC 668 Quantitative Research in Educational Technology
Dr. Seungoh Paek
January 29, 2014
Tonight’s Agenda
Announcements Feedback & Discussion on 611 vs. 668 Statistical Software Laulima vs. Weebly
Continuing Week 2 Experimental Research Methodology
Introduction to Statistics Designing & Conducting Educational
Technology Research
Feedback on 668 Course
Research Methodology Course
611 provides an overview of quantitative & qualitative research methods used in Educational Technology Research.
668 focuses on quantitative research methods used in Educational Technology Research.
667 focuses on qualitative research methods used in Educational Technology Research.
611 vs. 668
There will be some overlap since all three courses are research methodology classes. – The topics that overlap are ones that will be discussed and
reinforced throughout your doctoral program. – Understanding research methodologies is a major
component of doctoral work.
668 has a statistics component along with a focus on quantitative research methodology.
The focus of ETEC research will be different.
1. We will focus more on activities and application than on theoretical overview.
Course StructureSTATISTICS ETEC Research
Introduction to Statistics Introduction to Quantitative Research
Statistical Procedures Research Problem
Descriptive Statistics Literature Review
Frequency, Mean, Variance, etc. Research Methods
Inferential Statistics Research Design
Correlations Data Collection
Linear Regression Data Analysis
t-test Report Results
Analysis of Variance (ANOVA) Discussion & Presentation
2. We will focus on experimental design.
• Read experimental research articles.• Discuss experimental research design.• Conduct small-scale experimental
research.
Research Approaches
Quantitative approaches
Qualitative approaches
Survey research
Causal-comparative
research
Correlational research
Experimental research
Narrative research Phenomenological
research Ethnographic research Case study research Grounded theory research
Research Approaches
Quantitative approaches
Qualitative approaches
Survey research
Causal-comparative
research
Correlational research
Experimental research
Narrative research Phenomenological
research Ethnographic research Case study research Grounded theory research
Questions/Comments?
Statistical software Update
Laulima vs. Weebly
Continuing Week 2
Experimental Research Methods
Experiment in the lab?
~ Thorndike (1911)
Experimental Research
Experimental research is interested in the relationships between variables.
Experimental research provides strong evidence for causal interpretations.
To do so, researchers often manipulate variables, called independent variables systematically.
The results or effects from the variation of independent variables become dependent variables.
“Experimental” Designs
R X1 O (R = randomly assigned subjects; X = treatment)
R X2 O (O = observation/outcome)
Should be:– Replicable: repeat with the same results in another setting– Generalizable, representative– Cumulative: observations from earlier experiment used as a
basis for new one– Causal: establishes cause and effect (predictive)
Types of Experimental Educational Technology Research
Random assignment of subjects (“Experimental”)
Nonrandomized & single-subject designs (Quasi-experimental)
Quasi-Experimental
X1 O (X = treatment)
X2 O (O = observation/outcome)
Randomness is approximated through pre-tests to ensure “equivalence”
Group Activity II
Quasi-Experimental Research
Mayer, R. E., & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of educational psychology, 90(2), 312-320.
Dual-Processing Theory of Working Memory
a) Working memory includes an auditory working memory and a visual working memory.
b) Each working memory store has a limited capacity.
c) Meaningful learning occurs when a learner retains relevant information in each store, organize the information in each store into a coherent representation, and makes connections between corresponding representations in each store.
d) Connections can be made only if corresponding pictorial and verbal information is in working memory at the same time.
Research Design
Two experiments were conducted to test a prediction of a dual-processing theory of working memory.
Read the “Method” section for each Experiment: Group 1 = Exp 1; Group 2 = Exp 2.
Describe the procedure of each experiment. Summarize and report the findings of the
experiments.
Presentation
Dual-Processing Theory of Working Memory
Experiment 1 & 2 Results
Questions/Comments?
Yippee! I’m in statistics.
Statistics or Sadistics?:
It’s Up to You
Why this course…
Statistics for People Who (Think They) Hate Statistics – Follows an approach that is:
• Un-intimidating• Informative• Applied• Even a little fun!
Statistics: What It Is (and Isn’t)
Statistics describes “a set of tools and techniques that is used for describing, organizing, and interpreting data.”
Statistics is a problem-solving process that seeks answers to questions through data:– Ask a Question– Collect Appropriate Data– Analyze Data – Organize & Summarize– Interpret the Results
Success in this Course A few hints for successful completion of this
course– There are no dumb questions– How do you know statistics is hard?– Don’t skip lessons!! – Form a study group– Ask questions– Work through the exercises in each chapter– Practice, Practice, Practice– Look for real-world applications– Browse– HAVE FUN!!
Some websites with data
Substance Abuse & Mental Health Data Archive (SAMHDA) http://www.icpsr.umich.edu/SAMHDA/
Data2010… the Healthy People 2010 database http://wonder.cdc.gov/data2010/ftpselec.htm
http://www.lib.umich.edu/govdocs/stats.html Global Distribution of Poverty
http://sedac.ciesin.columbia.edu/povmap/
Descriptive or Inferential?
What is Descriptive Statistics?– Used to organize and describe the characteristics of a
particular data set• Example: the average age of everyone in this class!
What is Inferential Statistics?– Used to make inferences from your “sample” to the
“population”• Example: comparing the mean age of students taking this
course to average age of all students in an introductory statistics course
Basic Measurement Scales: Why Measurement?
You need to know that the data you are collecting represents what it is you want to know about.How do you know that the instrument you are using to collect data works every time (reliability) and measures what it is supposed to (validity)?
Scales of Measurement
Measurement is the assignment of values to outcomes following a set of rules
There are four types of measurement scales– Nominal– Ordinal– Interval– Ratio
Nominal Level of Measurement
Variables with characteristics that fits one and only one category
Name the attributes comprising a variable
Mutually exclusive categories such as male or female, Caucasian or African-American, etc.
Least precise level of measurement
Ordinal Level of Measurement
Next step beyond naming the attributes Characteristics being measured are ordered,
arranged in some order: from low to high, more to less, etc.
Rankings such as #1, #2, #3; liberal, middle-of-the-road, conservative
You know that a higher rank is better, but not by how much
Interval Level of Measurement
Variables having standard intervals of measurement
Intervals along the scale are equal to one another (same)
Lack a genuine zero point E.g. IQ, oC (Celsius) Interval & ratio scales usually lumped
together (in PSPP – called Scale)
Ratio Level of Measurement
Intervals along the scale are equal to one another (same)
Characterized by the presence of absolute zero on the scale
E.g. Age, oK (Kelvin scale) Distance between 10 yrs old and 20 yrs old
= Distance between 60 yrs old and 70 yrs old
Things to Remember
Any outcome can be assigned one of four scales of measurement
Scales of measures have an order The “higher” up the scale of measurement,
the more precise the data More precise scales contain all of the
qualities of the scales below it. Whenever possible, choose highest level –
ratio or interval scale
Part IIigma Freud & Descriptive
Statistics
Chapter 2 Means to an End:
Computing and Understanding Averages
What you will learn in Chapter 2
Measures of central tendency Computing the mean for a set of scores Computing the mode using the mode and
the median for a set of Selecting a measure of central tendency
Measures of Central Tendency
The AVERAGE is a single score that best represents a set of scores
Averages are also know as “Measure of Central Tendency”
Three different ways to describe the distribution of a set of scores…– Mean – typical average score– Median – middle score– Mode – most common score
Computing the Mean
Formula for computing the mean
“X bar” is the mean value of the group of scores
“” (sigma) tells you to add together whatever follows it
X is each individual score in the group The n is the sample size
XX
n
Things to remember…
N = population n = sample Sample mean is the measure of central
tendency that best represents the population mean
Mean is VERY sensitive to extreme scores that can “skew” or distort findings
Average means the one measure that best represents a set of scores– Different types of averages– Type of average used depends on the question
Weighted Mean Example
List all values for which the mean is being calculated (list them only once)
List the frequency (number of times) that value appears
Multiply the value by the frequency Sum all Value x Frequency Divide by the total Frequency (total n size)
Computing the Median
Median = point/score at which 50% of remaining scores fall above and 50% fall above.
NO standard formula– Rank order scores from highest to lowest or
lowest to highest– Find the “middle” score
BUT…– What if there are two middle scores?– What if the two middle scores are the same?
A little about Percentiles…
Percentile points are used to define the percent of cases equal to and below a certain point on a distribution– 75th %tile – means that the score received is at
or above 75 % of all other scores in the distribution
– “Norm referenced” measure• allows you to make comparisons
Computing the Mode
Mode = most frequently occurring score NO formula
– List all values in the distribution– Tally the number of times each value occurs– The value occurring the most is the mode
Democrats = 90Republicans = 70Independents = 140 – the MODE!!
– When two values occur the same number of times -- Bimodal distribution
When to Use What…
Use the Mode – when the data are categorical
Use the Median – when you have extreme scores
Use the Mean – when you have data that do not include extreme
scores and are not categorical
Glossary Terms to Know
Average Measures of Central Tendency
– Mean• Weighted mean• Arithmetic mean
– Median• Percentile points• Outliers
– Mode
Part IIigma Freud & Descriptive
Statistics
Chapter 2 Viva la Difference:
Understanding Variability
Why Variability is Important
Variability– how different scores are from one particular
score• Spread• Dispersion
So…variability is really a measure of how each score in a group of scores differs from the mean of that set of scores.
Measures of Variability
Three types of variability that examine the amount of spread or dispersion in a group of scores…– Range – Standard Deviation– Variance
Typically report the average and the variability together to describe a distribution.
Computing the Range
Range is the most “general” estimate of variability…
Two types…– Exclusive Range
• R = h - l
– Inclusive Range• R = h – l + 1
(Note: R is the range, h is the highest score, l is the lowest score)
3
2
l h
Computing Standard Deviation
Standard Deviation (SD) is the most frequently reported measure of variability
SD = average amount of variability in a set of scores
What do these symbols represent?
Why n – 1?
The standard deviation is intended to be an estimate of the POPULATION standard deviation…– We want it to be an “unbiased estimate”– Subtracting 1 from n artificially inflates the SD…
making it larger
In other words…we want to be “conservative” in our estimate of the population
Computing Variance
Variance = standard deviation squared
Standard Deviation or Variance
While the formulas are quite similar…the two are also quite different.– Standard deviation is stated in original units– Variance is stated in units that are squared
Glossary Terms to Know
Variability– Range– Standard deviation
• Mean deviation• Unbiased estimate
– Variance
Statistics Poker game
SPSS
Using SPSS
Using SPSS
Some websites with data
Substance Abuse & Mental Health Data Archive (SAMHDA) http://www.icpsr.umich.edu/SAMHDA/
Data2010… the Healthy People 2010 database http://wonder.cdc.gov/data2010/ftpselec.htm
http://www.lib.umich.edu/govdocs/stats.html Global Distribution of Poverty
http://sedac.ciesin.columbia.edu/povmap/
Designing & Conducting Educational Technology Research
Discussion
Share Your Thoughts
What is good research? How do you know? Why is it so hard to do good research in
Educational Technology?
Questioning the Questions of Instructional Technology Research
Is I.T. research socially relevant? Can I.T. research be socially relevant? Should I.T. research be socially relevant?
~ Reeves (1995)
Ed. Tech. Research Goals
Theoretical Predictive Interpretivist Postmodern Development Action
Theoretical Goals Focus on explaining
phenomena through logical analysis and synthesis of principles and results from other studies
EXAMPLE: Gagne’s theory of the conditions of learning
Predictive Goals Focus on determining
how education works by testing hypotheses related to theories of learning, teaching, performance, etc.
EXAMPLE: cooperative learning and control studies by Hooper, Temiyakarn, and Williams
Simon Hooper
Interpretivist Goals Focus on determining how
education works by describing and interpreting phenomena related to learning, teaching, performance, etc.
EXAMPLE: Delia Neuman’s observations of disabled children using commercial software
Delia Neuman
Postmodern Goals Focus on examining the assumptions underlying
educational programs with the goal of revealing hidden agendas and empowering disenfranchised minorities
EXAMPLE: Ann DeVaney’s analysis of IT in relation to race, gender, and power
Development Goals Focus on dual objectives of
developing creative approaches to solving problems and constructing reusable design principles
EXAMPLE: Cognition and Technology Group at Vanderbilt’s (John Bransford et al.) work with Jasper Woodbury problem-solving series
Evaluation/Action Goals Focus on describing,
improving, or estimating the effectiveness and worth of a particular program
EXAMPLE: Reeves and Laffey study of a problem-based learning engineering course at US Air Force Academy
Ed. Tech. Research Methods
Classification of ETRD (1989-94) & JCBI (1988-93) articles
More Recently
228 articles from the years of– 1985-1986– 1995-1996 – 2003-2004
~Jo, I. H. (2005)
Problem of Pseudoscience
Specification error Lack of linkage to robust theory Inadequate literature review Inadequate treatment implementation Measurement flaws Inconsequential outcome measures Inadequate sample size Inappropriate statistical analysis Meaningless discussion of results
Reeves’ Suggested Solution
Focus on making education work better Developmental research situated in schools
with real problem
Good Development ResearchDesign-Based Research Prerogative
Goals of designing learning environments and theories are intertwined
Development and research occur in continuous cycles
Research on designs leads to sharable theories relevant to practitioners
Research must account for how designs function in authentic settings
Development of accounts relies on methods that connect actions to outcomes
Ed Tech Research that Makes a Difference (Roblyer, 2005)
“need a more organized and persuasive body of evidence on technology’s benefits to classroom practice”
Ed tech research problems/challenges– Weak research design– Fragmented, uncoordinated studies– Poor methods-research questions match– Badly written reports– “ubiquitous interactions”– “hard-to-do science”– Technology changes quickly
Pillars of Good Educational Research
1. The Significance Criterion
2. The Rationale Criterion
3. The Design Criterion
4. The Comprehensive Reporting Criterion
5. The Cumulativity Criterion
New Ed Tech Research Agenda
Research to establish relative advantage Research to improve implementation
strategies Research to monitor impact on societal
goals Research that monitor & report on common
uses & shape desired directions
Design Research Example “Authentic Learning in
Interactive Multimedia Environments.”
Ph.D. dissertation by Jan Herrington at Edith Cowan University in Australia.
Supervised by Professor Ron Oliver.
Winner of AECT Young Researcher of the Year in 1999.
Outcome Practitioners Desired
New teachers will use a wider variety of assessment methods in their student teaching experience and eventual practice.
Learning Environment Design
Identified the critical characteristics of a situated learning model.
Developed an interactive multimedia learning environment based on those characteristics.
Situated Learning Model Herrington 1997
Provide an authentic context reflecting the way the knowledge will be used in real-life
Provide authentic activities Provide access to expert performances and
the modeling of processes Provide multiple roles and perspectives Support collaborative construction of
knowledge
Situated Learning Model Herrington 1997
Promote reflection to enable abstractions to be formed
Promote articulation to enable tacit knowledge to be made explicit
Provide coaching and scaffolding at critical times
Provide for integrated assessment of learning within the tasks.
Research Methodology
Mixed methods Videotaped preservice teachers
using program Interviewed teachers and their
supervisors in schools during student teaching practicum
Findings Problem Solved:
– Novice teachers acquired advanced knowledge while engaging in higher order thinking
– New knowledge and skills applied in practicum
Design Principles:– Situated learning model is
a successful design model for eLearning
Group Research Project Possible Group Research Project Topic
7-part Model for Conceptualizing Quantitative Ed Tech Research
1. Select a Topic2. Identify the Research Problem3. Conduct a Literature Review4. State the Research questions and hypotheses5. Determine the Research Design6. Determine the Methods7. Identify Data Analysis Procedures
Selecting a Topic
Identify general area of interest, focus Find something that you’re passionate
about Topic that would “make a difference” Something fun that you’re curious about
(hopefully!) Groups’ topic areas – future schools; cyber
charter school; tech integration; what else?
Start with Questions
As a researcher in the field of educational technology, what do you think is your task? In other words, what do you want to do as a ETEC researcher?
Is there an exemplary research paper that you have read about the field of educational technology?
What changes do you want to make to the field? Is there a contribution that you want to make? If so, in which area of this multidisciplinary area?
Identify the Research Problem
Determine problem/concern within your topic area
How important is the problem to the field How does it expand on existing knowledge Requires knowledge of the literature &
current research activities Bounce research problem/focus off others
What to do in Week 03
1. To get you started with your final research paper, discuss and post your group's tentative research topic/problem. (Forum Week 03)
2. Do the required readings for Week 04.Note: Readings are available for download on Laulima (Modules Week 04 Readings for Week 04)
What to do in Week 03
3. Read and present a experimental research article. Find a partner. Read one of the articles listed on the next slide. Prepare a 10 minute presentation of the
research paper (pretend you’re the authors of the article).
List of Readings for Multimedia Learning Principles
Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of educational psychology, 91(2), 358-368.
Mayer, R. E., Fennell, S., Farmer, L., & Campbell, J. (2004). A Personalization Effect in Multimedia Learning: Students Learn Better When Words Are in Conversational Style Rather Than Formal Style. Journal of Educational Psychology, 96(2), 389-395.
Mayer, R. E., & Johnson, C. I. (2008). Revising the redundancy principle in multimedia learning. Journal of Educational Psychology, 100(2), 380-386.
Holsanova, J., Holmberg, N., & Holmqvist, K. (2009). Reading information graphics: The role of spatial contiguity and dual attentional guidance. Applied Cognitive Psychology, 23(9), 1215-1226.