barry johnson - aptitude-achievement consistency analysis - dyslexia conference 2011

8/6/2019 Barry Johnson - Aptitude-Achievement Consistency Analysis - Dyslexia Conference 2011

1/62

1

Assessment: Meaning andInterpretation

Dr Barry Johnson

Dyslexia Action

1


2/62

2

The aim of this session is to update diagnosticassessors on developments within the area of ability-attainment discrepancy analysis and, inso doing, help them to be more confidentwith their day-to-day assessment practices.


3/62

3

Disclaimer

This presentation reflects the personalthoughts and opinions of Barry Johnson, HPCregistered, educational psychologist and notthe policies of Dyslexia Action.


4/62

4

Is an ability attainment-discrepancy

analysis for sp.l.d. diagnosis ...

Necessary?

Useful?Sufficient?

Irrelevant


5/62


6/62

6

P ianoplaying

skills

IQ


7/62

7

P ianoplaying

skills

IQ


8/62

8

H igh IQ H igh Level P iano Skills

C levercloggies

H igh IQ Low Level P iano Skills

Low IQ H igh Level P iano Skills

Low IQ Low Level P iano Skills


9/62

9


C levercloggies


Dyspianolexics




10/62

10


C levercloggies


Dyspianolexics


Idiotsavant-pianosyndrome



11/62

11


C levercloggies


Dyspianolexics

Low IQ

H igh Level P iano SkillsIdiotsavant-piano

syndrome

Low IQ

Low Level P iano SkillsDimbutnoisies


12/62

12

M usicalAptitude

P iano

playingskills

IQ F

ingerdexterity

M emoryforsounds

M usic sightreading


13/62

13

M usicalAptitude

P iano

playingskills

IQ F

ingerdexterity

M emoryforsounds

M usic sightreading


14/62

14

M usicalAptitude

P iano

playingskills

IQ F

ingerdexterity

M emoryforsounds

M usic sightreading


15/62

15

Aptitude-Achievement ConsistencyAnalysis

Aptitude-Achievement Consistency Analysis

Coo! What is this?

A contemporary alternative to the traditional

ability-attainment discrepancy model used tohelp in the diagnosis of dyslexia/sp.l.d.


16/62

16


What are the main criticisms of ability-attainmentdiscrepancy analysis?

(1) no close & clear link between assessment andteaching.

(2) use of arbitrary statistical cut-offs to judge degreeof discrepancy/clinical worth.

(3)over-reliance on old models of intelligence - g or

VIQ/Vis. IQ(4)Pragmatics time, statistical knowledge, etc.(5) Need for statistical knowledge (CCET plug!)


17/62

17


But, there is a claim that ability-attainment discrepancy analysisis still useful/needed in that it:

is clearly evident in daily practice - assessments and reports.

mentioned in DfES DSA Working Party Report.has an oblique reference in SEN Code of Practice.is offered in test manuals have ability-attainment look-uptables in Appendices.

is listed in Dyslexia Action s Consulting Psychologists qualityassurance requirements for report writing.Is used as an optional facility for Dyslexia Index - MartinTurner s D yslexia Portfolio (GL-Assessment).Is fed by consumer/cultural demand underachievement.


18/62

18


SEN Code of P ractice

7:39 H owever, academic attainment is not in itself sufficient

for LEAs to conclude that a statutory assessment is or is not necessary . An individual child s attainment must always beunderstood in the context of the attainments of the child s

peers, the child s rate of progress over time and, whereappropriate, expectations of the child s performance . A child sapparently weak performance may, on examination of theevidence, be attributable to wider factors associated with theschool s organisation. Careful consideration of evidence of low attainment may reveal good progress from a low base.


19/62

19


SpLD Working Group 2005/DfES Guidelines (DSA)

U nderlying Ability / Achievement differentials Although a discrepancy between underlying ability and attainment in literacy skills is not a diagnosticcriterion (Frederickson & Reason 1995, H owe 1997,

Miles 1996, Stanovich & Stanovich 1997, Siegel 1999), where such discrepancies do exist, they

provide further supporting evidence.


20/62

20

SpLD Working Group 2005/DfES Guidelines(cont.) Gathering information about underlyingability is an important component of assessment. Theassessment of verbal and non-verbal ability throwslight on the extent to which students are likely to beable to develop compensatory strategies, and informs specialist teaching intervention. The effect of SpLD on a student s learning can be evaluated moreeffectively when underlying ability is taken intoaccount.


21/62

21

SFE revised guidance for key delivery partners:

It is recommended that a full diagnosticassessment of a higher education student fordyslexia should contain the following


22/62

22

Evidence of any significant difficulties in reading, writing,

spelling and numeracy relevant to the standards in theseskills required in higher education.Evidence of a significant discrepancy between the abilitiesin reading, writing, spelling and numeracy and the level of those abilities expected of the student in terms of their

general intellectual ability .Evidence of cognitive disabilities or neurological anomalies(affecting memory, visual perception, phonologicalprocessing, including speed of processing or motorcoordination) which are likely to have adverse effects on

learning.Evidence that the learning difficulties identified are likely toaffect learning in higher education.


23/62

23


Some difficulties experienced by assessors:

Lack of a range of up-to-date cognitive ability andattainment tests.

Lack of co-normed ability-attainment look-up tables.Lack of statistical knowledge in order to compare

scores if tests used are not co-normed standard error of measurement, simple regression, standard error of estimate, confidence ranges , etc.


24/62

24


25/62

2 5


If one uses an IQ test and an attainment testthat have not been co-normed, thenregression to the mean needs to be takenaccount of when comparing discrepancyscores.Regression to the mean is affected by thecorrelation coefficients between the IQ andattainment tests. The lower the correlation,the bigger the regression effect.


26/62

2 6

S imple Regression formula

100100!d

X r y xy

IQ

Correlation coefficient for IQ & Attainment Test

Regressed score


27/62

2 7

Expected Regressed Scores

Test A Test B Test C Test Dr IQ TEST 0.4 0.6 0.8 0.9

IQ130 112 118 124 127

100 100 100 100 100

70 88 82 76 73


28/62

2 8


29/62

2 9

Note

correlationcoefficients


30/62

3 0

Noregression at

mean


31/62

3 1Significant regression at tails

of distributions


32/62

32


There are five dimensions of importance when applying A-ACAto a diagnosis of sp.l.d .:

(1) The examination of the degree to which a weakness inacademic achievement is consistent/positively correlated with related cognitive abilities/aptitudes.

(2) A-ACA is concerned in part with evaluation of consistency

among ability/aptitude & academic achievement of below normal limits - below the normal range is 85-115standardised scores.


33/62

33


(3) The evidence that the child has intact ability andacademic functioning in areas largely unrelated tothe presenting problem.

(4) The use of CHC model of cognitive abilities.(5) The use of aptitudes (narrow areas of cognitive

ability) as a conceptual bridge between broadcognitive abilities and academic skills. Interpretationis focused at the cluster level, yielding betterreliability.

(6) focus on exclusionary factors


34/62

34


B road abilities

Narrow aptitudes

Attainments


35/62

3 5


Rather than conceiving of cognitive abilities and academic achievements as mutually exclusive,

they may be better thought of as lying on anability continuum that has the most general types of abilities at one end and the most

specialized types of knowledge at the other end (Carroll, 1993).


36/62

3 6


You may have noticed that USA terminology isconfusing this is the reason

attainments/skills are commonly referred toas abilities. CHC theory has influenced theuse of such conceptual labels.


37/62

3 7


Brief summary of CHC Theory

The Cattell- Horn- Carroll Theory of cognitivedevelopment is a synthesis of the models byRaymond Cattell, John Horn, and John Carroll.Cattell postulated that there were two overallabilities people have: Crystallized intelligence andfluid intelligence. Crystallized intelligence reflectedabilities that were relatively static (such as learnedinformation) while fluid intelligence was morerelated to novel problem solving. (WR IT type model)


38/62

3 8


John Horn expanded this model by addingseven to nine (depending on your theoreticalorientation) broad abilities. They include:-


39/62

3 9

CrystallizedIntelligence

(Gc)

F luid Reasoning(Gf)

AuditoryP rocessing

(Ga)

P rocessing Speed

(Gs)

Short-Term

M emory(Gsm)

Long-Term

Retrieval(Glr)

Visual-SpatialP rocessing

(Gv)

Reading/Writing ability

(Grw)

QuantitativeReasoning

(Gq)


40/62

4 0


After analyzing past years of data, John Carroll cameup with his own set of broad abilities. He alsooffered a three-stratum theory of cognitivedevelopment.

Stratum III represents g or overall intelligence.Stratum II represents the broad abilitiesStratum I represents the narrow abilities grouped

under the broad abilities (for example, the broadconstruct of Gsm includes the narrow abilities of working memory and memory span.)


41/62

4 1


John Horn and John Carroll agreed tosynthesize their theories (Carroll s threestratum theory with Horn s broad abilities).The result is the CHC theory of cognitivedevelopment.


42/62

42

G e n e r a lSequent ia l

R e a s on ing

Induct ion

Q ua n t i t a t i veR e a s on ing

P ia ge t i a nR e a s on ing

Speed of R e a s on ing

F l u i dI n t e l l i g e n c e

( G f )

M a th . K now .

M a th . A ch .

Q u a n t i t a t i v eK n o w l e d g e

( G q )

L a ngua geDe ve l op .

L e x i c a lKnowl .

L i s t e n ingA b i l i t y

G e n e r a l I n f o .

Info.a b o u t C u l t u r e

G e n e r a l Sc i e n c e I n f o .

G e o g r a p h y A ch .

C o m m .Abil i ty

O r a lProduc t i on& Flu e nc y

G r a m .

Sensi t ivi ty

F o r e i g nL a ngua geProf i c i e nc y

F o r e i g nL a ngua geA p t i t u d e

C r y s t a l l i z e dI n t e l l i g e n c e

( G c )

R e a d i n gD e c o d i n g

R e a d i n gC o m p .

V e rb a lL a n g u a g eC o m p .

C lozeA b ili t y

S p e l l i n gA b ili t y

W r i t ing

A b ili t y

EnglishU sage K now ledg e

R e a d i n gS p e e d

R e a d i n g a n dW r i t i n g

( G rw )

M e m orySpan

L e a rn ingAbilities

S h o r t - Te r mM em o r y

( G s m )

V isualization

Spa t i a lR e la t i ons

V i sua lM e m o r y

ClosureSpeed

Flexibi l i tyof Closure

Spa t i a lScanning

Se r i a lP e r c e p t u a lIntegrat ion

L e ng thE s t ima tion

Pe rc e p t ua lI l lus ions

Pe rc e p t ua lA l te rna t ions

Im a ge ry

V i s u a lP r o c e ss in g

( G v )

P h o n . C d g . :A na l y s i s

P h o n . C d g . :S y n t h e s i s

Spe e c h Snd .Discrim.

R e s . t oA ud . S t im .Distortion

M e m o r yfor SoundPa t t e rns

G e n e r a lSnd. Discrim.

Te m p o r a lTra c k ing

M usicalD is c r im . &Judge m e nt

M ainta ining& JudgingRhyt hm

Snd- In t e ns i t yD ura t ionDiscrim.

Snd-Fre q .D i s c r im .

H e a r ing &S p e e c hThreshold

A bs o lu t eP i t c h

SoundL oc a l i z a t i on

A udi to r yP r o ce ss in g

( G a )

A ss o c .M em o r y

M n g f u l .M em o r y

F r e eR eca l lM e mo r y

Ide a t i ona lFl ue nc y

Assoc .F l ue nc y

E xp re s s ion a l

Fl ue nc y

N am ingFa c i l i t y

W ordF l u e n c y

FiguralF l u e n c y

FiguralFlexibi l i ty

Se ns i t i v i t y t o

Prob l e m s

O riginal i ty /C rea t ivi ty

L e a rn i ngA bil i t ies

L o n g - Te r mS t o r a g e &R et r i e v a l

( G lr )

PerceptualSpeed

R a te -o f -Test Taking

N umbe r Faci l i ty

P r o c e ss in gS p e e d

( G s )

SimpleRe a c t i on

Time

ChoiceRe a c t i onTime

Semant icProc e s s ingS p e e d

M e n t a lC ompa r i s onS p e e d

D eci s i o n /R e a c t i o n

T i m e / S p e e d( G t )


43/62

43


Cognitive ability and attainment test design in theUSA is aligning with the CHC model.

Up-to-date cognitive ability tests such as the W ISC IV

and WA IS IV have better alignment with CHC thanolder tests.XB A (Cross Battery Assessment) - commonlyreferred to in USA.

Lot of research collation linking aptitudes withattainments, feeding back into dynamic changes of CHC models.


44/62

44


Seems as though this process of taxonomy -linking aptitudes with attainments - isscientific, but is also helpful for moderationpurposes it promotes agreement in field. Isdynamic and changing according to ongoingresearch findings.


45/62

4 5


Following the evolution of CHC theory, cross batteryassessment emerged as a way to assess students. Theprinciple is to select tests from varied batteries which bestmatch the referral concern.

The examiner then puts the scores into a cross-batterytemplate, where the stratum II and I levels are aggregated(the subtests were matched with their stratum I and II counterparts by an expert consensus study.) Those scores arethen averaged.

Although combining subtests and averaging may not seemlike best practice, it has been noted to be an empiricallydefensible practice (See the FA Q section inhttp://www.crossbattery.com )


46/62

4 6


In essence, diagnostic assessors need:(1) a causal model/framework in their heads

that link underlying aptitudes with theapparent learning difficulties

(2) tests to sample these aptitudes(3) statistical means (confidence ranges will do)

by which they can observeconsistencies/inconsistencies across theseaptitudes and attainments.


47/62

4 7


(con.)(4) confidence to conclude the evidence(5) reduced reliance on a model that either drives the

perceived need to have a measure of untainted/pureg or where an ability-attainment discrepancy isperceived to be the main factor to drive thediagnosis of dyslexia/sp.l.d.

(5) assertiveness to apply and defend a normativeweakness model of disability/learning difficulties.


48/62

4 8


In USA there is a huge test industry: verypowerful and influential.

The big hitters of tests in this area are:


49/62

4 9


WJ III (2001) Abilities/WJ III W oodcock Johnson III Cognitive abilities/Cognitive

Achievement

SB5 (2003) Stanford Binet Intelligence ScalesKABC-II (2004)/KTEA II

Kaufman Assessment Battery for Children, Second

Edition (KABC-II

) /Kaufman Test of EducationalAchievement Second EditionDAS-II (2007)D ifferential Ability Scales Second Edition (DAS-II)


50/62

50

Given the historical predominance of the discrepancy model,evaluation of consistency may appear rather strange at first.

An aptitude score is comprised specifically of tests that are

most directly relevant to the development and acquisition of specific academic skills and thus is the best predictor of thecorresponding achievement area. For example, an individual with low reading ability and isolated cognitive deficits in oneor more aptitude areas related to reading achievement (e.g.,

phonological awareness, processing speed, short-termmemory) will most likely demonstrate consistency betweenscores of reading aptitude and reading achievement. Likewise,a high reading aptitude score would predict high readingachievement the two scores are more likely to be consistent

with each other rather than discrepant.Flanagan & Ortiz (2006)


51/62

51


Let us return to the A-ACA position on theimportance of the evaluation of consistency amongability/aptitude & academic achievement of below normal limits - below the normal range is 85-115standardised scores.

Note: this de-emphasises the importance of ability-attainment discrepancy when attainment scores areaverage or above-average.It also introduces a quantifiable criterion to facilitatedecision making in borderline cases and reduceoverall numbers of sp.l. being diagnosed.


52/62

52


(cont.) It is relevant to formal/legal definitions andinterpretations of disability in that a strictqualification/cut-off is given.

Evidence of average/above-average levels within theacademic domain of concern would lead to sp.l.d.not to be diagnosed. The null hypothesis stops at thefirst stage. Hunting for low scores or largediscrepancies between scores (massprofiling/Bonferroni problem) is discouraged.


53/62


54/62


55/62


56/62

56


What would all this look like? Would there be muchchange to our assessment paradigms and practices?

Not as great as you think, particularly with better

availability of co-normed up-to-date tests that mergeand span the ability-aptitude-attainment range.Even without these, cross-assessment batteryselection would help, provided assessors usedslightly more sophisticated statistics to help them .

We do have many research findings that havehighlighted aptitude-attainment links to literacydevelopment neatly summarised in, for example, :


57/62

57


Snowling, M.J. & Hume, C. (2010). Evidence-based interventions for reading and languagedifficulties: creating a virtuous circle . British J.of Ed. Psychology, 81, 1-23


58/62

58


What would reports look like? Not that muchdifferent, but advantages are that thediagnostic process would be transparent ,shared and therefore more consistent , andwould facilitate moderation processes to beapplied in cases of borderline cases

particularly where external criteria were notpresent (such as DSA) .


59/62

59


Megan reports having difficulties with single word readingrecognition. I found that she has weakness with her phoneticcoding (narrow Ga), naming facility (narrow Glr) and speed of

processing ( Gs) which are probably causally liked to her confirmed below average single word reading skills. In that she also has average abilities within the broad ability areas of general knowledge, fluid reasoning and attainment area of numerical operations, I would conclude that she has a specificlearning difficulty

Below-average aptitude-achievement consistency

Average/above average ability and/or attainment


60/62

60


Blodwyn, a H .E. student, is concerned that she has aspecific learning difficulty in that she believes she has weak spelling skills. H owever, my assessment demonstrates that her spelling skills lie within the average range for her age and that her other core literacy skills lie either within the averageof above average ranges. In that her core literacy skills arenot at below average levels, the presence of a sp.l.d. in thearea of concern can be discounted

Below-average aptitude-achievement consistency x

Average/above average ability and/or attainment


61/62

61


Ieuan has below-average levels for both reading aptitudeand core reading skill levels for his age. H e does not haveaverage or above-average aptitude/ability levels in other areas such as quantitative reasoning, memory and vocabulary knowledge. I therefore conclude that he does not have sp.l.d.but has general learning difficulties

Below-average aptitude-achievement consistency

Average/above average ability and/or attainment x


62/62

barry johnson - aptitude-achievement consistency analysis - dyslexia conference 2011

Documents