Folie 1Graduate surveys as an outcome evaluation
Presentation EAIR Forum 2009, July 23-26, Vilnius, Lithuania
Dr. René Krempkow
(former project leader of Freiburg Graduate Surveys, University of Freiburg)
Institut für Forschungsinformation und Qualitätssicherung
Institut für Forschungsinformation und Qualitätssicherung
iFQ
Outline
Measuring outcomes of higher education – Influencing factors of professional success
Exemplary results
Graduate Surveys
iFQ
When integrating graduate surveys in quality management:
Adequate interpretation of results only through comparison with other HEIs and more general national data (see Teichler 2003)
Account for individual characteristics through institution-specific and programme-specific surveys (see QM-literatur)
Nationwide “Core Questionnaire” with institution-/programme-specific adaptations
The Potential of graduate surveys for Quality Management I
Institut für Forschungsinformation und Qualitätssicherung
iFQ
Analyse data to identify determinants of professional success to feed back to Quality Management system
Unit of observation: study programmes
Account for specific context by interpreting data taking into account influencing factors (e.g. regional economy)
Account for specific initial conditions, e.g. % of student parents, socioeconomic background of students (Schomburg/Teichler, 1998)
The Potential of graduate surveys for Quality Management II
Institut für Forschungsinformation und Qualitätssicherung
Source: Schomburg/ Teichler (1998: 165)
Two HEI stand out positively
To what degree can differences be attributed to different starting conditions of HEIs? This will be discussed on the basis of Freiburg Graduate Surveys and Dresden Graduate Surveys.
The Potential of graduate surveys for Quality Management III
*
iFQ
Measuring Outcome with graduate surveys – what does professionell success depend on?
Multivariate analysis allows identifying factors influencing professionell success
This information is independant from (subjective) self-assessment of graduates (see Krempkow/Pastohr 2006)
Example: job experience, study times abroad and specific competences appear to be more important than study duration and grades – That leads to the question: Can and (if yes) What can HEIs do to improve professional success of graduate?
Multivariate (correlatory) analysis allows analysing relationships between professional success and aspects of study, including starting conditions and context!
Institut für Forschungsinformation und Qualitätssicherung
iFQ
Influencing factors on professional success
Objective and subjective criteria of professional success as indicators of HE outcomes (Teichler/Schomburg (1997: 248)
Objective indicators for professional success (e.g. income, hierarchical position)
Subjective indicators for professional success (e.g. job satisfaction, autonomy)
Objective indicators for transition HE-labour market (e.g. Time of job hunt)
Subjective assessment of goodness-of-fit HE – job (e.g. usefulness of study contents, employment adequacy).
Freiburg and Dresden Survey: Gross income, job satisfaction and employment adequacy fitting criteria for professional success.
Institut für Forschungsinformation und Qualitätssicherung
Basic model of influencing factors on professional success
INDIVIDUAL CHARACTERIS-TICS + SOCIO-ECONOMIC BACKGROUND
LEARNING OUTCOMES (COMPETENCES)
PROFESSIONAL SUCCESS /
PROFESSIONAL SUCCESS
Example of results: stand. Regression coefficients for income
Figure 3: Regression coefficients for the monthly income of current occupation; +/ */ ** = significant on the 10%-/ 5%-/ 1%-level (“-“ = variable was not included in the explanatory model)
Regression coefficients for monthly income of current occupation
Technical studies R2=.41 adj. R2=.00 n=105
Medical studies R2=.25* adj. R2=.11* n=270
Social sciences R2=.20* adj. R2=.08* n=316
Grade of matriculation standard
,317+
,051
,114
,093
-,141
,167
-,063
,104
-,130
,053
,111
,057
-,116
-,115
-,110
-,236
,066
-,089
Ability to critically rethink own ideas and ideas of others
,181
-,042
-,058
,063
-,323**
,011
-,043
Example of results: stand. Regression coefficients for income
Figure 3: Regression coefficients for the monthly income of current occupation; +/ */ ** = significant on the 10%-/ 5%-/ 1%-level (“-“ = variable was not included in the explanatory model)
Similar Results in Dresden Graduate Survey with the same analysis scheme, but the social background was more important
Regression coefficients for monthly income of current occupation
Technical studies R2=.41 n=105
Medical studies R2=.25* n=270
Social sciences R2=.20* n=316
Grade of matriculation standard
,063
-,323**
,011
iFQ
Interpretation of results (Freiburg & Dresden)
Influencing income: Previous job training, GPA (Secondary school) and gender, in Medicine: Personal contacts and career oriantation are important
Influencing job satisfaction:
Engineering: Tendendy for women to be less satisfied
Humanities: Higher self-assessed expertise and family orientation: lower job satisfaction
Influencing job adequacy:
Prospects for analysis?
iFQ
Determinants of professional success: Zeitschrift für Evaluation 1/2006 ( www.zfev.de ), S. 7-37, bzw. http://www.kfbh.de/downloads/Was_macht_Hochschulabsolventen_erfolgreich.pdf
Dresden Graduate Survey: www.kfbh.de/absolventenstudie
Example of results: stand. Regression coefficients
Institut für Forschungsinformation und Qualitätssicherung
2nd Example: Job Adequacy