National adaptations to main survey instruments and layout

verification

National Research Coordinators Meeting Windsor, June 2008

NRCMeetingWindsor

June 2008

Contents of presentation

• Principles of instrument adaptation

• Outline of adaptation review undertaken by the ISC

• National Adaptation Form (example)

• Problems identified in field trial

• Procedures for layout verification

NRCMeetingWindsor

June 2008

Adapting main survey instruments

• All instruments have to be adapted– including those for English-speaking

countries

• Adaptations should be kept to a minimum

• Adaptations have to be documented in National Adaptation Forms (NAF)

• There needs to be a consultation with the International Study Centre (ISC)

NRCMeetingWindsor

June 2008

Types of adaptations

• There are parts of instruments that have to be adapted– Provision in translation notes and NAF– Usually in carets (< >)

• There may be the need for adapting other items– This should only be done when absolutely

necessary!

• In some cases, NRCs may want to add items or categories– Should be discussed and agreed with ISC

NRCMeetingWindsor

June 2008

Purpose of National adaptation forms

• Necessary information for translation verification

• Reference for layout verification

• Facilitates data processing– Documenting any deviation from

international format

• Informs later data analysis

NRCMeetingWindsor

June 2008

Process of adapting instruments

• Completing NAF in cooperation with translator and reviewer (> Version I)

• Consultation about NAF with ISC (> Version II)

• Documenting any changes from translation verification (> Version III)

• Documenting any changes from layout verification (> Version IV)

NRCMeetingWindsor

June 2008

National adaptation forms (NAF)

• EXCEL format

• One general information spreadsheet

• Five (or six) adaptation forms– Cognitive test– Student background questionnaire– Student perceptions questionnaire– Teacher questionnaire– School questionnaire– Regional instrument (if applicable)

NRCMeetingWindsor

June 2008

Columns in NAF - 1

• Columns 1 to 3 should not be modified:– 1: International question numbers– 2: International question/category wording– 3: International variable or code

• Column 4 to 5 may have to be completed– Only when adding items or categories!

NRCMeetingWindsor

June 2008

Columns in NAF - 2

• Columns 6 to 7 have to be completed when marked in yellow:– 6: National wording of adaptation– 7: English translation of this adaptation

• Column 8 is reserved for any comments NRC would like to make– Useful for consultation when implementing

adaptations that are additional

• Column 9 is reserved for comments by the ISC during the consultation

NRCMeetingWindsor

June 2008

Adding items or categories

• Not acceptable for cognitive test• Requires discussion and agreement

with ISC• Only acceptable in exceptional

circumstances• MUST be documented in NAF!• For countries that participated in

CIVED no changes in adaptation/translation are permitted!

NRCMeetingWindsor

June 2008

NAF example

NRCMeetingWindsor

June 2008

International and national options

• International options will be included in NAF– NRCs should indicate whether they want to include the

optional question(s)

• National options should be documented in the NAF– Please list questions (with English translation) that should

be added

– A special spreadsheet will be included for this purpose

– National questionnaire parts should not be excessively long (otherwise follow-up sessions would be required)

NRCMeetingWindsor

June 2008

Assistance with NAF

• More detailed information at Data Management Seminar in Hamburg

• NRCs should contact the ISC in case of any problems with the completion of NAF

NRCMeetingWindsor

June 2008

Experience from the field trial - 1

• Generally, procedures for adaptation review worked very well

• Adds important element to quality assurance procedures

• Provides important information for– Translation verification– Data processing– Data analysis

NRCMeetingWindsor

June 2008

Experience from the field trial - 2

• Some improvements that were identified during this process– Adaptation notes needs to be complemented

based on experiences during adaptation review– There should be a space for recording

adaptations to general directions– There should be a space for recording national

items

• Some incomplete NAF were submitted which required re-submission– Reminder to national centres to document ALL

adaptations

NRCMeetingWindsor

June 2008

Layout verification

• Final step for instrument preparation before going into the field

• The layout verification is carried out at the International Study Centre (ACER)

• The layout verification is done AFTER translation verification– Assembly of test booklets once translation

verification results have been received

• ALL countries are required to go through layout verification BEFORE printing their instruments

NRCMeetingWindsor

June 2008

How to prepare for layout verification

• Finalise instruments based on advice from translation verification

• Assembling and layouting material according to sample instruments– available in PDF and WORD

• Prepare instruments in WORD and converting them into PDF

NRCMeetingWindsor

June 2008

What to submit?

• Countries should submit PDF files of:– Six test booklets– Student questionnaire– Teacher questionnaire– School questionnaire

• Version III of the NAF– including any updates after translation

verification

NRCMeetingWindsor

June 2008

What we will do

• Compare the layout of national instruments with the sample instruments

• Generally, no further checks of translations or adaptations– if spotted we will advise NRCs

• Results of layout verification sent back to national centres (EXCEL spreadsheet)

NRCMeetingWindsor

June 2008

After layout verification

• National centres should implement corrections as advised

• Sometimes the ISC may ask for re-submission in case of severe layout problems

• Final instruments should be submitted with data files to IEA DPC together with final NAF (version IV)

NRCMeetingWindsor

June 2008

Experience from the field trial

• Generally, procedures worked well• There were a few cases where instruments

had to be re-submitted• Due to the narrow timelines some

instruments had not been submitted for layout verification– National centres should ensure that their national

instruments go through ALL steps in the verification process (adaptation review, translation verification and layout verification)

NRCMeetingWindsor

June 2008

Questions or comments?