Overview of ICCS field trial

National Research Coordinators Meeting Windsor, June 2008

NRCMeetingWindsor

June 2008

Content of presentation

• Participation in field trial• General outcomes of field trial• Feedback from national centres

– Summary in NRC(June08)1.pdf

• Problems identified in field trial• Consequences of field trial

NRCMeetingWindsor

June 2008

Field trial outcomes

• Field trial data analysis– revealed that generally the instruments worked

well• Cognitive test well targeted to ability & reliable• Questionnaire scales reliable and fit structure

– informed the selection of item material for main survey

– confirmed general feasibility of on-line surveys for teacher and schools

• Review of implementation procedures– Generally procedures worked well– Confirmed the feasibility of using regional

instruments– Some possibilities for improvement &

streamlining

NRCMeetingWindsor

June 2008

Field trial participation

• 32 countries undertook field trial

• One country to field trial in June 2008

• 31 data sets included in international field trial analysis report

• Approximately– 19,000 students – 9,000 teachers– 700 schools

NRCMeetingWindsor

June 2008

School response rates

• 21 countries had school response rates of at least 85 percent BEFORE replacement

• 4 countries had school responses rates of at least 85 percent AFTER replacement

• Only 7 countries had school response rates below 85 percent

• Period for field trial close to the end of the year contributed to problems with school participation

NRCMeetingWindsor

June 2008

Student response rates

• Overall, about 90 percent of students participated within schools

• Response rates (slightly) below 85 percent were found in only two countries

NRCMeetingWindsor

June 2008

Teacher response rates

• Across countries, there was a median teacher response rate within schools of 90 percent

• When combined with school response rates the overall response rate for the teacher survey was 89 percent (median across countries)

• In 10 out of 31 countries the overall response rate was below 75 percent

NRCMeetingWindsor

June 2008

Results of NRC Survey

• Overall satisfaction with manuals

• Main problems encountered:– Convincing schools to participate– Explaining sampling procedures– Implementing some test administration

procedures– Reaching a high student and teacher

participation within schools

NRCMeetingWindsor

June 2008

Issues raised regarding survey instruments

• Length of student test and questionnaires (7 countries)

• Questions in instruments seen as too difficult for students of this age (4 countries)

• General directions too long and repetitive (5 countries)

NRCMeetingWindsor

June 2008

Issues raised regarding the teacher sampling design

• Listing of all teacher teaching in target grade seen as a problem

• Two national centres saw the procedures as a major obstacle for school participation

• Data about teachers’ month/year of birth seen as inappropriate

NRCMeetingWindsor

June 2008

Issues raised regarding teacher participation

• Teachers not teaching CCE saw questionnaire as irrelevant to them

• Two national centres viewed questionnaire as too long

• Teacher participation more difficult to obtain than student participation

NRCMeetingWindsor

June 2008

Issues raised regarding regional modules

• Problems with layout of European instrument (part 2 being upside-down and back-to-front)

• Problems with administration and length of general directions

NRCMeetingWindsor

June 2008

Issues raised regarding manuals and software

• More than 80% found the manuals “very helpful” but suggestions to:– Simplify and shorten manuals – Refine software tools for sampling and

data entry

• Need to improve the scoring guides for coding of open-ended questions

NRCMeetingWindsor

June 2008

Issues identified by the ISC

• Need for improvement of adaptation notes

• Provision for countries to record their national adaptations to general directions

• Inclusion of checklist in manual for submission for verification

NRCMeetingWindsor

June 2008

Looking ahead

• Reduction in assessment time– Testing time now 45 minutes– Student questionnaire time now about 35

minutes– Regional instruments shorter– Some reduction in teacher and school

questionnaires

• Reduction in teacher sample size– Cluster size of 15 for main survey– On average, no more than 15 teachers in target

population at schools

NRCMeetingWindsor

June 2008

Expert advice

• Project Advisory Committee Meeting 7-8 June here in Windsor

• Discussion of outcomes and proposals

• Some changes to the proposed instruments following PAC advice– PAC suggestions will be included in

presentations

NRCMeetingWindsor

June 2008

Questions or comments?