adaptive methods research methods fall 2010 tamás bőhm
TRANSCRIPT
![Page 1: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/1.jpg)
Adaptive Methods
Research Methods
Fall 2010
Tamás Bőhm
![Page 2: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/2.jpg)
Adaptive methods
• Classical (Fechnerian) methods: stimulus is often far from the threshold
inefficient
• Adaptive methods: accelerated testing– Modifications of the method of constant stimuli
and method of limits
![Page 3: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/3.jpg)
Adaptive methods
• Classical methods: stimulus values to be presented are fixed before the experiment
• Adaptive methods: stimulus values to be presented depend critically on preceding responses
![Page 4: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/4.jpg)
Adaptive methods
• Constituents– Stepping rule: which stimulus level to use next?– Stopping criterion: when to finish the session?– What is the final threshold estimate?
• Performance– Bias: systematic error– Precision: related to random error– Efficiency: # of trials needed for a specific precision;
measured by the sweat factor
![Page 5: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/5.jpg)
Notations
Xn stimulus level at trial n
Zn response at trial nZn = 1 detected / correct
Zn = 0 not detected / incorrect
φ target probabilityabsolute threshold: φ = 50%
difference threshold: φ = 75%2AFC: φ = 50% + 50% / 2 = 75%4AFC: φ = 25% + 75% / 2 = 62.5%
xφ threshold
![Page 6: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/6.jpg)
Adaptive methods
• Classical methods: stimulus values to be presented are fixed before the experiment
• Adaptive methods: stimulus values to be presented depend critically on preceding responses
Xn+1 = f(φ, n, Zn, Xn, Zn-1, Xn-1,…, Z1, X1)
![Page 7: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/7.jpg)
Adaptive methods
• Nonparametric methods:– No assumptions about the shape of the
psychometric function– Can measure threshold only
• Parametric methods:– General form of the psychometric function is
known, only its parameters (threshold and slope) need to be measured
– If slope is also known: measure only threshold
![Page 8: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/8.jpg)
Nonparametric adaptive methods
• Staircase method (aka. truncated method of limits, simple up-down)
• Transformed up-down method• Nonparametric up-down method• Weighted up-down method• Modified binary search• Stochastic approximation• Accelerated stochastic approximation• PEST and More Virulent PEST
![Page 9: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/9.jpg)
Staircase method
• Stepping rule:Xn+1 = Xn - δ(2Zn - 1)– fixed step size δ– if response changes:
direction of steps is reversed
• Stopping criterion:after a predetermined number of reversals
• Threshold estimate: average of reversal points(mid-run estimate)
• Converges to φ = 50% cannot be used for
e.g. 2AFC
![Page 10: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/10.jpg)
Transformed up-down method
• Improvement of the simple up-down (staircase) method
• Xn+1 depends on 2 or more preceding responses– E.g.1-up/2-down or 2-
step rule:• Increase stimulus
level after each incorrect response
• Decrease only after 2 correct responses
• φ = 70.7%
• Threshold:mid-run estimate
• 8 rules for 8 different φ values(15.9%, 29.3%, 50%, 70.7%, 79.4%, 84.1%)
5 10 15 200
10
20
30
40
50
trial
stim
ulu
s le
vel
reversal points
![Page 11: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/11.jpg)
Nonparametric up-down method
• Stepping rule: Xn+1 = Xn - δ(2ZnSφ - 1)
– Sφ: random numberp(Sφ=1) = 1 / 2φp(Sφ=0) = 1 – (1 / 2φ)
– After a correct answer:stimulus decreased with p = 1 / 2φstimulus increased with p = 1 - (1 / 2φ)
– After an incorrect answer: stimulus increased
• Can converge to any φ ≥ 50%
![Page 12: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/12.jpg)
Nonparametric up-down method
5 10 15 200
10
20
30
40
50
trial
stim
ulu
s le
vel
![Page 13: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/13.jpg)
Weighted up-down method
• Different step sizes for upward (δup) and downward steps (δdown)
1
downup
![Page 14: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/14.jpg)
Modified binary search
• ‘Divide and conquer’• Stimulus interval
containing the threshold is halved in every step(one endpoint is replaced by the midpoint)
• Stopping criterion: a lower limit on the step size
• Threshold estimate:last tested level
• Heuristic, no theoreticalfoundation
Figure from Sedgewick & Wayne
![Page 15: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/15.jpg)
Stochastic approximation
• A theoretically sound variant of the modified binary search
• Stepping rule:– c: initial step size– Stimulus value increases for correct responses,
decreases for incorrect ones– If φ = 50%: upward and downward steps are equal;
otherwise asymmetric– Step size (both upward and downward) decreases
from trial to trial
• Can converge to any φ
)(1 nnn Zn
cXX
![Page 16: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/16.jpg)
Stochastic approximation
5 10 15 200
10
20
30
40
50
trial
stim
ulu
s le
vel
![Page 17: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/17.jpg)
Accelerated stochastic approximation
• Stepping rule:– First 2 trials: stochastic approximation– n > 2:
step size is changed only when response changes (mreversals: number of reversals)
• Otherwise the same as stochastic approximation
• Less trials than stochastic approximation
)(21
nreversals
nn Zm
cXX
![Page 18: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/18.jpg)
Accelerated stochastic approximation
5 10 15 200
10
20
30
40
50
trial
stim
ulu
s le
vel
![Page 19: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/19.jpg)
Parameter Estimation by Sequential Testing (PEST)
• Sequential testing:– Run multiple trials at the same stimulus level x– If x is near the threshold, the expected number of
correct responses mc after nx presentations will be around φnx
the stimulus level is changed if mc is not in φnx ± w
– w: deviation limit; w=1 for φ=75%
• If the stimulus level needs to be changed:step size determined by a set of heuristic rules
• Variants: MOUSE, RAT, More Virulent PEST
![Page 20: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/20.jpg)
Adaptive methods
• Nonparametric methods:– No assumptions about the shape of the
psychometric function– Can measure threshold only
• Parametric methods:– General form of the psychometric function is
known, only its parameters (threshold and slope) need to be measured
– If slope is also known: measure only threshold
![Page 21: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/21.jpg)
Parametric adaptive methods
• A template for the psychometric function is chosen:– Cumulative normal– Logistic– Weibull– Gumbel
x
xp
exp1
1)(
![Page 22: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/22.jpg)
Parametric adaptive methods
• Only the parameters of the template need to be measured:– Threshold– Slope
x
xp
exp1
1)(
![Page 23: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/23.jpg)
Fitting the psychometric function
1. Linearization (inverse transformation)of data points
• Inverse cumulative normal (probit)• Inverse logistic (logit) p
pp
1log)logit(
2 4 6 80
0.2
0.4
0.6
0.8
1
2 4 6 8
-8
-6
-4
-2
0
2
4
![Page 24: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/24.jpg)
Fitting the psychometric function
2. Linear regression
3. Transformation of regression line parameters
0 2 4 6 8 10
-8
-6
-4
-2
0
2
4
6
2 4 6 80
0.2
0.4
0.6
0.8
1
X-intercept & linear slope Threshold & logistic slope
![Page 25: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/25.jpg)
Contour integration experiment
0 7.5 11.5 15.5 19.5 23.5
0
0.25
0.5
0.75
1
orientation jitter ( )
ratio
of
corr
ect
resp
onse
s
3=15.14
D = 2
slope = -0.6
0 7.5 11.5 15.5 19.5 23.5
0
0.25
0.5
0.75
1
orientation jitter ( )
ratio
of
corr
ect
resp
onse
s
3=20.29
D = 65
slope = 0.3
![Page 26: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/26.jpg)
Contour integration experiment
0 7.5 11.5 15.5 19.5 23.5
0
0.25
0.5
0.75
1
orientation jitter ( )
ratio
of
corr
ect
resp
onse
s
1=17.43
2=18.65
3=19.37
4=19.44
5=20.23
0 7.5 11.5 15.5 19.5 23.5
0
0.25
0.5
0.75
1
orientation jitter ( )
ratio
of
corr
ect
resp
onse
s
1=13.84
2=14.84
3=15.14
4=15.44
5=14.43
5-day perceptual learning
![Page 27: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/27.jpg)
Adaptive probit estimation
• Short blocks of method of constant stimuli• Between blocks: threshold and slope is
estimated (psychometric function is fitted to the data) and stimulus levels adjusted accordingly– Assumes a cumulative normal function
probit analysis
• Stopping criterion: after a fixed number of blocks• Final estimate of threshold and slope:
re-analysis of all the responses
![Page 28: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/28.jpg)
Adaptive probit estimation
• Start with an educated guess of the threshold and slope
• In each block: 4 stimulus levels presented 10 times each
• After each block: threshold ( ) and slope ( ) is estimated by probit analysis of the responses in block
• Stimulus levels for the next block are adjusted accordingly– Estimated threshold and slope
applied only through correctionfactors inertia
rrrrrrrr
cc ,3
,3
,
)ˆ(1 rrr corr
)ˆ(1 rrr corr
r̂ r̂/1
rr/1
![Page 29: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/29.jpg)
Measuring the threshold only
• Function shape (form & slope) is predetermined by the experimenter
• Only the position along the x-axis (threshold) needs to be measured
• Iteratively estimating the threshold and adapting the stimulus levels
• Two ways to estimate the threshold:– Maximum likelihood (ML)– Bayes’ estimation
• QUEST, BEST PEST, ML-TEST, Quadrature Method, IDEAL, YAAP, ZEST
![Page 30: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/30.jpg)
Maximum likelihood estimation
• Construct the psychometric function with each possible threshold value
• Calculate the probability of the responses with each threshold value (likelihood)
• Choose the threshold value for which the likelihood is maximal (i.e. the psychometric function that is the most likely to produce such responses)
)|,()( thresholdZXpthresholdL
- - - +
- - + +
- + + +
![Page 31: Adaptive Methods Research Methods Fall 2010 Tamás Bőhm](https://reader035.vdocuments.mx/reader035/viewer/2022062417/551c0c19550346b24f8b51a7/html5/thumbnails/31.jpg)
Bayes’ estimation
• Prior information is also used– Distribution of the threshold in the population
(e.g. from a survey of the literature)– The experimenter’s beliefs about the threshold
)()|,()( thresholdpthresholdZXpthresholdL
a priori distribution of the threshold
values of the psychometric
functions at the tested stimulus
levels