introduction to alis - amazon web serviceswsassets.s3.amazonaws.com/ws/tlr/files/downloads/... ·...
TRANSCRIPT
Introduction to Alis
Dr Robert Clark
ALIS Project Manager
Ensuring Fairness
Principles of Fair Analysis :
1. Compare ‘Like’ with ‘Like’
2. Appropriate Baseline
3. Reflect Statistical Uncertainty
The Analysis
.
Subject X
4 5 6 7 8
Linear Least Squares Regression
Outcome
Subject X
10
8
6
4
2
0
+ve VA ve VA
0 2 4 6 8
Baseline Residuals
Regression Line (…Trend Line, Line of Best Fit)
Outcome = gradient x baseline + intercept
Correlation Coefficient (~ 0.7)
Measuring ValueAdded – An Example
National Trend
Result
Subject A
‘Average’ Student A
B ve
Alf Bob C
+ve Chris D
Subject B
E
U
Low Ability Average Ability High Ability
Baseline Score
The position of the national trend line is of critical importance
Some Subjects are More Equal than Others….
Gra
de
A
B
C
D
E
Physics
Maths
Psychology
Sociology
Latin
Photography
English Lit
>1 grade
C B A A*
Average GCSE
Principle of Fair Analysis No1 : Compare ‘Like’ with ‘Like’
Some Subjects are More Equal than Others …
Performance varies between subjects, thus analysing and predicting each subject individually is essential.
e.g. Student with Average GCSE = 6.0
Subject Choices Predicted Grades
Maths, Physics, Chemistry, Economics
C, C, C/D, C/D
Sociology, Communication Studies,
Drama, Media
B, B/C, B/C, B/C
Standardisation of Residuals
• (Raw) Residuals can be used to examine an individual’s performance
• Standardised Residuals are used to compare performance of groups
• Standardised Residuals are independent of year or qualification type
• For a class, subject, department or whole institution the Average Standardised Residual is the ‘ValueAdded Score’
• Standardised Residual = Residual / Standard Deviation (National Sample)
• When using Standardised Residuals then for an individual subject
1 where N = number of results in the group Standard Error =
N
(for combinations of subjects consult the relevant project)
• 95% Confidence Limit = 2.0 x Standard Error
• 99% Confidence Limit = 2.6 x Standard Error
• 99.7% Confidence Limit = 3.0 x Standard Error
Subjects Covered…
• A / AS Levels
• Applied A / AS levels (including dual award)
• International Baccalaureate
• BTec Nationals (Diploma, Certificate, Award)
• CACHE DCE
• OCR Nationals
• ifs Diploma / Certificate in Financial Studies
• Limited pool of level 2 (BTec First)
How to Administer the
Project
1. Submit a registration form (Y11 May onwards….) • We need this before we can process any data • We need this even if you are registering as part of a consortium • Choose Basic / Full (Basic + Attitudinal surveys) and whether you wish to do baseline test
2. Submit student details – ‘Registration Spreadsheet’ (Y12 Mid Sept onwards….) • This gives us student name details, GCSE scores and the subjects they are studying • We always need this even if the students are sitting a baseline test • Send spreadsheet once students are confirmed on courses (i.e. not first day of term….)
3. Organise baseline testing – ‘Adaptive Test’ (End Y11 June 15th onwards….) • This can happen before, at the same time as or after sending us the registration
spreadsheet (2 above) • Student details appear in ‘Check List’ on web site • Early prediction are available for students with Adaptive Test scores as soon as they
appear in the Check List. This function is removed one Alis has generated offical predictions (pdf reports).
• Don’t forget to click ‘Testing Complete’ once you have finished testing your students.
4. Prediction Reports Generated • Prediction reports, Intake Profiles, Adaptive Test data (IPR) • Reports created after receipt of Registration Spreadsheet
� Guarenteed turnaround 4 weeks � Normal deliverable turnaround 2 weeks
• When adaptive test data is ready (‘Testing Complete’ clicked), repots are updated.
5. Maintain Data • Keep reports up to date by using the Subject Editor on the Alis+ secure website to add
and remove students from subject registrations and request updated feedback
6. Submit Entries Data (Y12 & Y13 March / April) • For institutions offering A / AS options, submit EDI entries files to Alis
7. Entries data Matched and Check lists issued (Y12 & Y13 May July) • These need to be completed to ensure complete matching of candidate numbers to
names held by Alis to ensure all EDI exam results are successfully processed in August
8. Results Collection (Y12 & Y13 August) • Submit A / AS results to Alis via EDI Results Files • Submit Other quals (IB, BTEc etc) to Alis using results spreadsheet (can opt to submit A /
AS data in spreadsheet as well instead of EDI files) • Submit results as soon after results day as possible
9. Preliminary VA Feedback (Beginning of September) • Preliminary feedback generated by 1st Monday in September. Prompt return of results in
August leads to early feedback
• Trend data not fixed, values may be subject to change
10. Definitive VA Feedback (End of Spetember • Trend data locked and feedback generated. Letter & CD sent to schools / colleges.
11. Maintain Data • Update results data (missing grades, withdrawls, remarks, appeals etc) using the Results
Editor on the Alis+ secure website and request updated feedback.
Typical Timeline
Y11 Sept Oct Nov Dec Jan Feb March April May June July Aug
Prediction Reports (+Y13)
Registration Form
CABT Early Preds
15th
Y12 Sept Oct Nov Dec Jan Feb March April May June July Aug
Registration Form
Registration SSheet CABT (+ Early Preds)
Entries Collection & Matching
Matching Checklists R
Value Added Feedback
Y13 Sept Oct Nov Dec Jan Feb March April May June July Aug
Results Collection
Entries Collection & Matching
Matching Checklists R
Value Added Feedback
Y14 Sept Oct Nov Dec Jan Feb March April May June July Aug
Results Collection
Baseline Assessment
Choice of Baseline
• Average GCSE Score
• CABT (Computer Adaptive Baseline Test)
Why 2 Baselines ?
Why 2 Baselines ?
Average GCSE correlates very well to Alevel / IB etc, but by itself is not sufficient….
• What is a GCSE ?
• Students without GCSE ?
• Years out between GCSE & Alevel ?
• Reliability of GCSE ?
• Prior ValueAdded ?
Principle of Fair Analysis No2 : Appropriate Baseline
The Effect of Prior Value Added
Beyond Expectation In line with Expectation Below Expectation
+ve ValueAdded 0 ValueAdded ve ValueAdded
Average GCSE = 6 Average GCSE = 6 Average GCSE = 6
Do these 3 students all have the same ability ?
Appropriate Baseline
• Do students with the same GCSE score from feeder schools with differing valueadded have the same ability ?
• How can you tell if a student has underachieved at GCSE and thus can you maximise their potential ?
• Has a student got v.good GCSE scores through the school effort rather than their ability alone ?
• How will this affect expectation of attainment in the Sixth Form ?
• Can you add value at every Key Stage ?
Baseline testing provides a measure of ability that (to a large extent) is independent of the effect of prior treatment.
Computer Adaptive Baseline Test (CABT)
• Test performed online – results automatically transmitted to CEM.
• Minimal installation / setup required if any.
• Adaptive – difficulty of questions changes in relation to ability of student.
• Efficient – no time wasted answering questions that are far too easy or difficult.
• Wider range of ability
• Less stressful on students – more enjoyable experience than paper test.
• Less demanding invigilation.
• Test designed to be completed in 1 hour or less.
• No materials to courier
In 2009 / 2010 over 65,000 students sat this test in Alis
To try it out… www.intuproject.org/demos
Feedback
Intake Profiles
Intake Profiles (Historical)
IPR...
Studying : Maths Physics Chemistry Biology
Full Alis 2009 Demo School (999)
Banana, Brian
Banana, Brian
?
Prediction Reports
Probability of achieving
each grade
Expected Grade
Which predicted grades are the most appropriate for ths student ?
Prediction Reports
Subject Report
Alis
Value Added Feedback
Burning Question :
What is my ValueAdded Score ?
Better Question :
Is it Important ?
Principle of Fair Analysis No3 : Reflect Statistical Uncertainty
Value Added Feedback…
SPC Chart
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year
Subject Summary Current Year
Subject Summary 3 Year Average
2002 2003 2004
Year
A2English Literature
Statistical Process Control (SPC) Chart
Average
Standardised
Residual
0.60
0.48
0.36
0.24
0.12
0.00
0.12
0.24
0.36
0.48
0.60
0.60
0.48
0.36
0.24
0.12
0.00
0.12
0.24
0.36
0.48
0.60 2008 2009 2010
Year
Student Level Residuals (SLR) Report
Scatter Plot
A2 – English Literature
General Underachievement ?
Student Level Residuals (SLR) Report
Scatter Plot
A2 – English Literature
Too many U’s ?
Other things to look for…
Why did these students do so badly ?
Why did this student do so well ?
How did they do in their other subjects ?
Summary of Process
• Examine Subject Summary
• Determine ‘interesting’ (i.e. statistically significant) subjects
• Look at 3 year average as well as single year
• Look at trends in ‘Interesting Subjects’
• Examine student data – SLR Report, scatter graphs
• Identify students over / under achieving (student list in SLR or Paris)
• Any known issues ?
• Don’t forget to look at over achieving subjects as well as under achieving
• Phone / EMail ALIS when you need help understanding / interpreting the data / statistics !
Attitudinal Surveys
There is more to school / college than exams….
• Student attitudes • Student Welfare & Safety • Nonacademic activities • Support • Social and personal development
Self Evaluation (Every Child Matters)
Full ALIS
Attitudinal Feedback...
5 5
4 4
3 3
2 2Attitude to Institution 1 1
Attitude
Scale
Attitude
Scale
3.8 3.9 3.8 4.0 3.8 3.7
4.1
3.5 3.8 4.0
3.7 3.7 3.8 4.0
3.2
5 5
4 4
3
Attitude to Subject 3
2 2
1 1
2.1
3.5 3.9
3.6
3.0
3.6 3.4
2.9
3.4 3.8 3.6
2.8
3.7 3.8 3.7
40 40
32 32
24 24
16 Private Tutors 16
8 8
0 0
Percent
5
17 19
9 13
20
13
33 33
00000 3
Extended Attitudes
Teaching and Learning Processes Teaching and Learning Processes (999) CCCCEEEEMMMM CCCCEEEE NNNNTTTT RRRREEEE
A2Physics
Average Value
1.0 2.0 3.0 4.0 5.0 6.0
Producing original w ork
Receiving individual help f rom the teacher
Receiving help f rom another student
Giving help to another student
Listening to another student presenting w ork
Presenting w ork to the class
Working in pairs
Researching a topic
Making use of IT (computers)
Using audio or visual materials
Doing practical w ork
Using duplicated notes (handouts)
Making ow n notes f rom lessons
Having notes dictated
Having discussions in groups
Having class discussions led by the teacher
Reading
Preparing essays or reports
Working f rom previous exams (exam condns)
Working f rom previous exams (w ith help)
Working through examples
Having a topic presented by the teacher 4.0
4.5
2.4
2.1
1.6
3.8
3.1
2.4
3.4
4.4
3.3
3.5
3.2
3.9
3.0
2.7
2.0
1.8
2.4
2.3
2.9
3.0 [2.9]
[3.2]
[2.6]
[2.6]
[1.8]
[1.8]
[3.1]
[2.7]
[2.6]
[2.7]
[3.7]
[3.5]
[3.7]
[3.7]
[2.5]
[3.3]
[3.3]
[1.6]
[2.4]
[3.2]
[4.5]
[3.8]
[Background Value]
1.0 2.0 3.0 4.0
A verage Value
5.0 6.0
The numbers on the graph ref er to the f ollowing scale...
1 = never or almost never
4 = about once a fortnight
2 = about once a term
5 = about tw ice a week
3 = about once a term
6 = about every lesson
The background is the average value of the 3638 students studying this subject in the Full ALISProject who expressed an opinion. The bars represent the average value of the 54 students studying this subject in your institution who expressed an opinion.
Teaching and Learning Processes A2Physics
SEF Survey
• Extent of Bullying
• Extent of Racism
• Extent of Sectarianism
• Healthy Lifestyles
• How Well do Learner's Make a Positive Contribution to the Community
• How Well do Learner's Prepare for Their Future Economic Well Being
• Other Health and Safety Issues
• School's/College's Action on Bullying
• School's/College's Action on Racism
• School's/College's Action on Sectarianism
• School's/College's Action on Sexual Harassment
• Spiritual, Moral, Emotional and Cultural Development
To them out… www.intuproject.org/demos
School’s / College’s Action on Sexual Harassment
Summary
Alis History : • Alis began in 2003 (started life called COMBSE)
• Developed in partnership with schools by professional educational researchers
• After spreading locally in the North East, Alis grew rapidly nationwide, largely through decisions by individual schools and colleges to subscribe
• Alis is part of CEM which is affiliated to the School of Education at Durham University.
• Research by CEM acknowledged by Durham in contributing significantly to the international research reputation of the School of Education.
• Alis – developed in an educational context, by educational professionals for use by educational professionals.
Alis Coverage : • Approx 1700 school / colleges anually
• > 50% UK Alevels anually
• A/AS; IB; BTec National; OCR National; Cache DCE
• Developing BTec First; GCSE Resit
Alis Provides… Baseline Tests : • GCSE not always an appropriate or reliable measure of
ability
• GCSE Scores depend on KS4 valueadded performance
• Alternative baseline test available
• Provides predictions and valueadded analysis independent of performance at prior key stage
Alis Provides… Predictions :
• Predictions targeted at the individual subject (on average, students with similar GCSE scores get different grades in different Alevel subjects)
• Predictions from GCSE and Alis baseline test (how reliable is GCSE as a measure of ability ? Does the student have GCSE’s ?)
• Predictions at 50th and 75th percentile
• Chances Data (what is the probability of achieving grades different to those predicted?)
• Standardised Scores from the baseline tests including section breakdown (IPR Report) – what are the student’s strengths & weaknesses ?
Alis Provides… Value Added :
• All VA scores are specific to the student and each individual subject
• Reports at school, subject and student level.
• Current and historical trend data
• Three sets of reports available:• Subject Level (Whole cohort) • Syllabus Level (Whole Cohort) • Subject Level (Specific to your school type)
• VA available from GCSE and from the Alis baseline test
• All data shown against appropriate confidence limits
• Analysis available from beginning of September
• Consortia / area / LA analysis available
Alis Provides… Attitudes :
• In depth subject related attitudinal survey
• In depth student welfare survey covering: • Extent of Bullying • Extent of Racism • Extent of Sectarianism • Healthy Lifestyles • How well do learners make a positive contribution to the community • How well do learners prepare for their future economic well being • School / College action on bullying • School / College action on racism • School / College action on sectarianism • School / College action on sexual harassment • Spiritual, moral, emotional and cultural development
• Can provide evidence to use in Self Evaluation
To them out… www.intuproject.org/demos
Alis data can be used:
• To support teaching and learning � School Band Profile Graphs � IPR Data � ‘Predictive’ data for target setting and monitoring � Paris software for data analysis and on course monitoring
• To aid target setting and monitoring � Use reliable predictive data (e.g. Alis data) � Use professional judgment, including knowledge of the student � Consider school/department expectations and ethos � Give consideration to previous value added data where it is available (e.g.
Alis data)
• For Value Added analysis
• For Self Evaluation
Inset provision is available on any aspect of Alis to support any of the above issues.
