iccs 2009 idb seminar – nov 24-26, 2010 – iea dpc, hamburg, germany iccs 2009 design, databases...
TRANSCRIPT
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
ICCS 2009Design, Databases and Materials
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Content
Survey DesignStatus of PreparationInternational Database
Structure: File Types, Countries, Variables, Missing DataDocumentation: Codebooks, Instruments, IDB User Guide
Seminar MaterialAnalysis (Do‘s and Dont‘s)
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Content
Survey DesignStatus of PreparationInternational Database
Structure: File Types, Countries, Variables, Missing DataDocumentation: Codebooks, Instruments, IDB User Guide
Seminar MaterialAnalysis (Do‘s and Dont‘s)
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Design – Objective
ICCS 2009 is intended to investigate the ways in which young people are prepared to undertake their roles as citizens.It will report on:
Student achievement (civic knowledge)Student disposition and attitudesTechers‘ teaching and class management practicesSchool governance and climateNational context of civic and citizenship educationAsian, European and Latin American regional issues
4
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Design – Surveys
Student SurveyStudent Achievement BookletsStudent Questionnaire
Teacher SurveyTeacher Questionnaire
School SurveySchool Questionnaire
National Context SurveyNational Context Questionnaire
5
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Design – Target Populations
Students: All students enrolled in the grade that represents eight years of schooling, counting from the first year of ISCED Level 1, providing the mean age at the time of testing is at least 13.5 years (usually grade 8). Students older than 17 years are not part of the target population.Teachers: All teachers teaching regular school subjects to students of the target grade (regardless of the subject or the number of hours taught) during the ICCS testing period, and have been employed at school since the beginning of the school year.
6
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Design – Participating Countries
38 countries from around the world
7 Source: Schulz et al., 2010, ICCS 2009: International Report, IEA, Amsterdam
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Design – Sample Structure
Stratified multi-stage probability sampling designRandom samples were required for each populationMinimum sample sizes:
150 schools1 intact class per school, aiming for an achieved sample size of 3,000 target grade students15 target grade teachers (or all if number of target grade teachers was less or equal to 20)
8
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Design – Sample Structure
More than 5.300 schools, 140.000 target grade students, and 62.000 target grade teachers
9
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Design – Sample Structure
Reporting of data from the student survey need to be cautious for some countries and reflect the following issues:
HKG and NLD did not meet sampling requirements.BFL, CZE, DNK, NZL, NOR, CHE, THA met guidelines for sampling participation rates only after replacement schools were included.ENG nearly satisfied guidelines for sample participation only after replacement schools were included.GTM, KOR, PRY surveyed the same cohort of students but at the beginning of the next school year.SVK national desired population does not cover all of international desired population.
10
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Design – Sample Structure
Reporting of data from the teacher survey need to be cautious for some countries and reflect the following issues:
AUT, BFL, DEN, ENG, HKG, LUX, NZL, NOR, and CHE did not meet sampling requirements.CZE, SWE, THA met guidelines for sampling participation rates only after replacement schools were included.IRL nearly satisfied guidelines for sample participation only after replacement schools were included.SVK national desired population does not cover all of international desired population.GRC and NLD data from the teacher survey could not be reported.
All the above is annotated in the international reports.
11
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Design – Sample Structure
More information about the sampling design will be detailed in chapters 2 and 10 of the ICCS 2009 Technical Report
12
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Design – CIVED Link
Data from 17 countries could be linked to data from the IEA CIVED study from 1999.To cover a similar target population, four countries surveyed an additional grade (grade 9): GRC, NOR, SWE, SVN
13
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Content
Survey DesignStatus of PreparationInternational Database
Structure: File Types, Countries, Variables, Missing DataDocumentation: Codebooks, Instruments, IDB User Guide
Seminar MaterialAnalysis (Do‘s and Dont‘s)
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Status of Preparation
Data itself confirmed by countriesData Processing at the IEA DPC
Data preparation doneDerived variables addedWeights added
15
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Status of Preparation
Changes since last data send out (v3.1, April 2010)Current data version 3.2Scale SCSTUDOP (Principals' perceptions of student opportunities to participate in community activities) in School data file (ICG) changed so that category “not offered at school” is now coded “0” instead of receiving a missing code3 new scales added to Student Questionnaire data file (ISG):
NISB: National Index of Socioeconomic BackgroundPAREDYRS: Highest educational level in approximate years of schoolingEXPEDYRS: Expected years of schooling (recoded from SISCED)
16
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Status of Preparation
Timeline for future data send out(November 23 sent out to NRCs)January 2011: put on IEA website for public download, including Technical Report and IDB User Guide
17
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Content
Survey DesignStatus of PreparationInternational Database
Structure: File Types, Countries, Variables, Missing DataDocumentation: Codebooks, Instruments, IDB User Guide
Seminar MaterialAnalysis (Do‘s and Dont‘s)
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
International Database - Content
Country specific filesData files (labeled SPSS format, SAS format, national variables where applicable)
DocumentationCodebooks (WinDEM format)Technical ReportIDB User Guide
Data and documentation will be published in January 2011 on IEA website: www.iea.nl
19
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
International Database - Files Types
ICG School Questionnaire FileITG Teacher Questionnaire FileISG Student Questionnaire FileISA Student Achievement FileISR Student Reliability FileISE European Module Student FileISL Latin American Module Student FileISS Asian Module Student File
20
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
International Database - Files Types
Additional grade data (GRC, NOR, SWE, SVN)JSG Student Questionnaire FileJSA Student Achievement FileJSR Student Reliability FileJSE European Module Student File
21
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Cases included:School questionnaires that were returned(CPART = 3) Teacher questionnaires that were returned (TPART = 3)AND where schools met the minimum within school teacher participation rate of 50%Student data that were returned (SPART = 3)AND where schools met the minimum within school student participation rate of 50%NOTE: A student was counted participating if either achievement or questionnaire data was available (regional module data only was not sufficient)Indicator variable: INICS09 = 1
22
International Database - Files Types
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Note that the student files and also the teacher file may contain cases from schools where the school questionnaire has not been returned.
23
International Database - Files Types
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
International Database - Data Structure
From the surveyID variables (record, scorer, puncher, linking) and, for students, booklet rotationListing and tracking variables (e.g., ITPARTR) Response variables and, for students, constructed response scores
24
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
International Database - Data Structure
From the data processing and weighting stageParticipation (SPART, TPART, CPART) and adjudication flags (INICS09)Stratification information (IDSTRATE/I)Design and estimation weightsVariables needed for variance estimation (JKZONES, JKREPS)Meta data (VERSION, IDCNTRY, DPCDATE)
25
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
With the information in the NAFs national adaptations could be reflected and data were recoded to fit international database structure in the majority of casesFull back-translation into English will be important for readers of the report and also for secondary research (especially if more than only country specific terms < > were adapted)
International Database - National Adaptations
26
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Main goal: document the comparability or non-comparability of country dataDocumentation of important cultural and other national adaptations that affect the meaning of a variable
International Database - National Adaptations
27
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
If national adaptations made recoding of data necessary, this has been documented in the National Adaptations ReportsIf international comparability is not achievable, data for international variable(s) were set to not administered (but country data were still available as national variable)If possible, data for not administered variable(s) were gang punched to a certain value
E.g., ITLANG has been gang punched to ‘1’ if only one language was used for all instruments in a country
International Database - National Adaptations
28
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
National Adaptations Reports will inform and support secondary analysisWill be published as part of the ICCS 2009 complementary publication (IDB User Guide)
International Database - National Adaptations
29
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
International Database - Variables
Identification variables (ID…) Tracking variables (IT…)Survey instrument variablesWeighting variablesScale scores
30
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Identification Variables
Unique IdentifiersIDSCHOOL
School ID4 digitsUsed in all files
31
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Identification Variables
IDCLASSClass ID6 digitsFirst 4 digits identify schoolLast two digits used to identify class within schoolUsed in the student files (ISG, ISA, ISR, ISE, ISL, ISS)
32
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Identification Variables
IDTEACH Teacher ID6 digitsUsed in teacher file (ITG)IMPORTANT NOTE: Teachers are NOT linked to classes and NOT directly linked to students!
33
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Identification Variables
IDSTUDStudent ID8 digitsFirst 6 digits include the corresponding class (IDCLASS)Used in student data files
34
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Tracking Variables
CPARTTPARTSPART Final Participation IndicatorsITLANG Administration Language (1, 2, ..., 5)IDCNTRY Country ID for ReportingVERSION Data Sendout Version (currently 32)DPCDATE Date Exported (currently 20101116)
35
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Variables
IC2G01A…IC2G23 survey instrument variables in school data file (ICG)IT2G01A... IT2G29I survey instrument variables in teacher data file (ITG)
36
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Variables
CI2COM1…CI106M1 survey instrument variables in student achievement data file (ISA)IS2G03... IS2G18G student background variables in student questionnaire data file (ISG)IS2P19A... IS2P36F student perception variables in student questionnaire data file (ISG)
37
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Variables
EST01A…EST12 student achievement variables in European module data file (ISE)ESP01A…ESP13D student perception variables in European module data file (ISE)LST01…LST16 student achievement variables in Latin American module data file (ISL)LSP01A…LSP12I student perception variables in Latin American module data file (ISL)ASP01A…ASP08E student perception variables in Asian module data file (ISS)
38
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Survey Variables
All survey instrument variables that have been adapted for national purposes, have been recoded to fit the international schemeNational variables will be provided to country, but not be part of international database
39
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Variables for Weighting and VE
School data fileTOTWGTC Final school weight – school studyJKZONEC Jackknife zones – school studyJKREPC Jackknife replicate code – school study
40
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Variables for Weighting and VE
Teacher data fileTOTWGTT Final teacher weight – teacher studyJKZONET Jackknife zones – teacher studyJKREPT Jackknife replicate code – teacher study
41
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Variables for Weighting and VE
Student data filesTOTWGTS Final student weight – student studyJKZONES Jackknife zones – student studyJKREPS Jackknife replicate code – student study
42
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Content Knowledge Variables
PV1CIV…PV5CIV: Civic knowledge plausible valuesCentered to mean of 500 and standard deviation of 100 scale points (for equally weighted countries)NWLCIV: National civic knowledge score (WLE)
43
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Scales & Derived Variables
For derived indices, see the list of indices(C:\ICCS2009\Documentation\ICCS2009_Indices.xls)Example: RELIG
Students’ belonging to a religionDummy coded to ‘No religion’ (code 0) – ‘Religion’ (code 1)
44
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Scales & Derived Variables
For derived scales, see the scale documentation(C:\ICCS2009\Documentation\ICCS2009_Scales.doc)Example: POLDISC
Students' discussion of political and social issues outside of schoolIRT WLE scores with mean of 50 and standard deviation of 10 for equally weighted countriesItems included
IS2G13AIS2G13DIS2G13FIS2G13G
45
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Missing Values
Special codes represent missing dataCodes contain information on nature of missing dataCodes and levels of missing data should be checked before usage
46
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
(SPSS)
9
99
...
Missing Values – “Omitted”
No Response
47
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
(SPSS)
7
97
...
Missing Values – “Invalid”
Invalid Response
48
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Question not reached
Question omitted
(SPSS)6
96…
Missing Values – “Not Reached”
(SPSS)2
49
(SPSS)9
99…
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
(SPSS)
System-missing
.
Missing Values – “Not Administered”
• Nationally not administered variables
50
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
(SPSS)
5
95
...
Question 2 does not apply according to
filter question
Missing Values – “Logically Not Applicable”
51
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Codebooks
Contain information related to the structure of file types including the
SourceFormatDescriptive labelsResponse option codes for all variables
One codebook file is provided for each of the instruments/ file typesAvailable in .txt format that can be opened with text editorAll dictionary information can be found directly in the SPSS file
52
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Codebooks
53
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Content
Survey DesignStatus of PreparationInternational Database
Structure: File Types, Countries, Variables, Missing DataDocumentation: Codebooks, Instruments, IDB User Guide
Seminar MaterialAnalysis (Do‘s and Dont‘s)
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Seminar Material
AgendaSeminar agenda
CodebooksFor each of the file types (ICG, ITG, ISA, ISG, ISR, ISE, ISL, ISS)
DataSPSS data (one file per country and file type)
55
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Seminar Material
DocumentationList of IndicesScale documentationDraft chapters of IDB User Guide
All these is DRAFT information (do not cite or quote). Information will be published in the IDB User Guide.
56
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Seminar Material
InstrumentsSchool QuestionnaireTeacher QuestionnaireStudent QuestionnaireRegional Module Questionnaires
European (questionnaire part only)Latin American (questionnaire part only)Asian
PresentationsAll presentations of this seminar
57
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Seminar Material
Software7-Zip: compressing and archiving files (freeware)Adobe Reader 9: reading PDF docuementsIEA IDB Analyzer: analyzing IEA data
WorkSPSS script needed for seminar hands-on training
AddIns_ICCSSupport material and examples for hands-on training of statisticial significance tests
HLM_TrainingSupport material and examples for hands-on training of HLM sample analysis
58
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Content
Survey DesignStatus of PreparationInternational Database
Structure: File Types, Countries, Variables, Missing DataDocumentation: Codebooks, Instruments, IDB User Guide
Seminar MaterialAnalysis (Do‘s and Dont‘s)
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Analyzing ICCS 2009 Data
ICCS 2009 is a survey and collected observational dataWe know what schools, teachers, and students characteristics are, what they think, and the context in which this occursWe have current cross-sectional information, while the reason or effect might have occurred a while backThe data has invaluable descriptive powerWe can make statements about associations and correlations, not causation
60
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Analyzing ICCS 2009 Data
ICCS 2009 is NOT an experimentWe do not control the assignment of students and teachers to ‘treatment’ groupsEvents have already happened and we record what has happenedWe can not establish causality, or direct effects
61
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Analyzing ICCS 2009 Data
School variables ONLYUse ICG file
Teacher variables ONLYUse ITG file
Student variables ONLYUse ISA, ISG, ISE, ISL, and/or ISS file
62
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Analyzing ICCS 2009 Data
Student variables using school level attributes (combined)
Merge corresponding files (i.e. disaggregate school variables to students)Supported by the IEA IDB AnalyzerUse the student weight (TOTWGTS)Mind that there might be students of a school of which the school questionnaire was not returned or vice versa, so watch levels of missing data
63
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Analyzing ICCS 2009 Data
School variables using aggregated student level characteristics
Not supported by the IEA IDB AnalyzerIf at all, use with caution (ecological fallacy)
64
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Analyzing ICCS 2009 Data
Teacher variables using school level attributes (combined)
Merge corresponding files (i.e. disaggregate school variables to teachers)Supported by the IEA IDB AnalyzerUse the teachers weight (TOTWGTT)Mind that there might be teachers of a school of which the school questionnaire was not returned or vice versa, so watch levels of missing data
65
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Analyzing ICCS 2009 Data
Teacher data should not be merged with student dataThere is no direct linkage between teachers and studentsTeachers teaching in the school a student is attending might have never seen that student or taught a lesson to the student
66
ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany
Thanks for your attention!