Quantitative data analysis:Planning an analytic strategy for your research

John Richardson

Probation BenchmarksThe project report should include the following:• A viable research question• A critical literature review which situates the proposed

research• A research proposal including an outline of proposed

method(s) and a critical justification for them• A work plan for the project.

Project Benchmarks, ctd.• The proposed methods should include methods of data

collection and methods of data analysis. • Whether you are carrying out qualitative or quantitative

research, you should know broadly how you are going to analyse your data before you collect them.

• The work plan for your project should include a realistic estimate of the time it will take you to do the analysis.

• My aim is to get you to think creatively about the kinds of analysis that might address your research problem.

Some terminology• Population: the complete set of individuals, objects or

scores that the researcher is interested in studying.• Sample: the subset of the population actually studied.• Parameter: a property of a population• Statistic: a property of a sample• Descriptive statistics: the use of statistics to describe the

data that have been obtained• Inferential statistics: the use of statistics to generalise

from properties of a sample to properties of a population.

Brush up your statistics• Here are the URLs for two PowerPoint presentations on

descriptive statistics and inferential statistics. • If you have done the MRes programme, you will have

seen them before but might want to refresh your memory. • If you haven’t done the MRes programme, they should be

useful in bringing you up to scratch with statistics.• Read the one on descriptive statistics first.

Brush up your statistics, ctd.• Descriptive statistics:http://learn.open.ac.uk/file.php/6071/Block_1_powerpoints/H809_Descriptives.ppt

• Inferential statistics:http://learn.open.ac.uk/file.php/6071/Block_1_powerpoints/H809_Inferential.ppt

Measurement scales• Check that you understand the measurement properties

of your data.• A nominal scale is simply a classification (e.g. gender).• An ordinal scale is a classification where the categories

can be ordered in some way (e.g. social class).• An interval scale is one where there are equal intervals

between the ordered categories (e.g. temperature).• A ratio scale is one where the equal intervals start at a

logical absolute zero (e.g. height and weight).

Examples of three kinds of data• Demographic characteristics• Performance measures• Self-report measures

I’m an educational researcher, and I’m therefore going to use examples drawn mainly from education. I will end up by giving a brief account of measures of effect size.

Demographic characteristics• Usually treated as nominal measurement scales (e.g.

gender, ethnicity)• In principle, age could be treated as a ratio

measurement scale (equal intervals and a logical zero). • But it is usually unnecessary to ask for exact age: the

response rate might be higher if you use age bands.• Treating age as an nominal measurement scale is more

likely to reveal nonlinear effects (e.g. people in their late 20s and early 30s are most likely to get good degrees).

Demographic characteristics, ctd.• How is demographic information collected? Self reports,

other reports, institutional records?• Participants may decline to provide sensitive data.• For instance, they may decline to assign themselves to

ethnic categories that seem inappropriate.• People with disabilities may fail to disclose them for fear

of being stigmatised.

Demographic characteristics, ctd.• Relationships among demographic characteristics take

the form of contingency tables. • For example, Richardson and Wydell (2003) explored

the prevalence of dyslexia in students at UK institutions of higher education.

• Here are their data with respect to the prevalence of dyslexia in different age bands and in men and women.

Demographic characteristics, ctd.• The raw data take the form of contingency tables. For

instance, the top half of previous slide is based on a table showing the numbers of students with dyslexia and with no reported disability in five different age bands.

• Similarly, the bottom half of the slide is based on a table showing the numbers of students with dyslexia and with no reported disability for men and women separately.

Demographic characteristics, ctd.• Contingency tables are most conveniently analysed

using chi-squared tests to see whether apparent relationships could have come about by chance. These tests are widely available in computer packages.

• They sometimes come with an option to use a correction for continuity. If you haven’t come across this before, don’t worry. If you have, don’t use it (Richardson, 1994).

Demographic characteristics, ctd.• In separate analyses, Richardson and Wydell showed

(a) that the prevalence of dyslexia declined with age and (b) that the prevalence of dyslexia was higher in men.

• The decline with age probably reflects the increased identification of dyslexia in children and young people over the last 30-40 years.

• Boys may be more vulnerable to dyslexia than girls. However, schools may just be more likely to identify dyslexia in boys than in girls.

Demographic characteristics, ctd.• Are the effects of age and gender independent of each

other, or is there an interaction between them?• This needs to be addressed by examining a multiway

frequency table: this is a contingency table showing the numbers of students with dyslexia and with no reported disability in each of the five age bands for men and women separately.

Demographic variables, ctd.• Multiway contingency tables can be analysed using

loglinear analysis. • This examines the relationship between two or more

categorical variables with any number of categories.• It generates likelihood ratio statistics that behave rather

like the chi-squared statistic.

Demographic variables, ctd.• Using loglinear analysis, Richardson and Wydell found

that there were significant effects of gender and age but there was a significant interaction between the effects.

• The gender difference was significant for younger students, but not for those aged 40 and over.

• This doesn’t support the idea that boys are more vulnerable (which should show up in all age bands).

• Instead, the way that schools have identified dyslexia over the last 40 years is more likely to identify it in boys.

Performance measures• The most common measures of academic performance

are marks, which can usually be regarded as falling on an interval measurement scale.

• The performance of different groups can be compared using Student’s t test (for two groups) or (univariate) analysis of variance (for two or more groups).

• These are both parametric tests that make assumptions about the properties (parameters) of the populations from which the samples are drawn.

Performance measures, ctd.• However, performance may instead be expressed in

terms of grades or degree classes, which are probably best regarded as falling on an ordinal scale.

• This requires the use of nonparametric tests which do not make assumptions about the properties of the populations from which the samples are drawn.

• The performance of different groups can be compared using the Mann-Whitney U test (for two groups) or the Kruskal-Wallis test (for two or more groups).

Performance measures, ctd.• Sometimes performance is reduced to a single binary

measure, such a completion/noncompletion, pass/fail, or good degrees/poor degrees (i.e. those awarded with first-class or upper second-class honours versus those with lower second-class or third-class honours.)

• This is clearly measured just on a nominal scale. If the predictor variables are also measured on a nominal scale, the data constitute a multiway frequency table.

Performance measures, ctd.• Richardson and Wydell compared the proportion of good

degrees awarded to students with dyslexia and students with no reported disability across the United Kingdom

• Of the students with dyslexia, 43.9% obtained good degrees; of the students with no reported disability, 53.6% obtained good degrees.

• Multiway frequency analyses showed that the difference was statistically significant and remained so even when the effect of other variables were taken into account.

Performance measures, ctd.• Another kind of analysis that can be used to investigate

the effects of predictor variables measured on any kind of measurement scale is logistic regression.

• This can be used in situations where the predictor variables are measured on an interval scale.

Performance measures, etc.• Logistic regression analysis expresses the results in terms

of odds ratios: the ratio between the odds of a certain outcome in two different groups.

• These vary from zero (the event never happens in the first group) to one (it is equally likely to happen in both groups) to infinity (it never happens in the second group).

Performance measures, ctd.• Richardson (in press) found that, across all universities,

(a) the odds ratio comparing the likelihood of Asian and White students obtaining good degrees was 0.50, and (b) the odds ratio comparing the likelihood of Black and White students obtaining good degrees was 0.33.

• In other words, the odds of an Asian student getting a good degree and the odds of a Black student getting a good degree are only 50% and 33%, respectively, of the odds of a White student getting a good degree.

Performance measures, ctd.• Logistic regression analysis showed that these odds

ratios increased to 0.71 and 0.60 when the effects of other background variables were taken into account.

• So about half of the disparity in the performance of different ethnic groups is attributable to background variables (mainly differences in entrance qualifications and in their choice of different academic subjects).

Self-report measures• Many educational researchers use questionnaires and

other self-report measures.• In some cases, it may be sufficient to analyse responses

to individual items, in which case they can be analysed accordingly as nominal-scale or ordinal-scale data.

• However, in many cases the responses to sets of items are totalled or averaged to obtain scores on particular scales (e.g. deep and surface approaches to studying).

Self-report measures, ctd.• A questionnaire is reliable to the extent that it would

yield consistent results if used repeated under the same conditions with the same participants.

• This is measured by various coefficients of reliability that vary between zero (reflecting total unreliability) and one (reflecting perfect reliability).

• The most common approach to measuring the reliability of questionnaire scales is to examine the consistency among the responses given to their constituent items.

Self-report measures, ctd.• The most widely used measure of internal consistency is

Cronbach’s (1951) coefficient alpha. • A common rule-of-thumb is that an adequate instrument

should yield values of coefficient alpha of at least 0.70. • Even if a questionnaire generally yields satisfactory

values, this needs to be checked in each new context.

Self-report measures, ctd.• Entwistle et al. (2000) devised the Revised Approaches

to Studying Inventory (RASI) to measure use of a deep approach, a strategic approach or a surface approach.

• Richardson (2006) adapted the RASI for use in distance education and obtained data from 1,462 students taking courses in the Open University’s Faculty of Technology.

• The values of coefficient alpha for the three main scales were: deep approach, 0.83; strategic approach, 0.79; and surface approach, 0.66.

Self-report measures, ctd.• An instrument is valid to the extent that it measures the

personal qualities or traits that it purports to measure.• The construct validity of an instrument is whether the

sets of items measure distinctive traits or constructs and whether these are related together in the expected way.

• This is investigated by means of factor analysis, which tries to identify hypothetical constructs in the patterns of relationships among the items or scales.

Self-report measures, ctd.• Confirmatory factor analysis tests whether a specified

factor structure fits a set of data. • In confirmatory factor analysis, one needs to specify in

advance (a) the number of factors, (b) the loadings of each of the individual items or scales on those factors, and (c) the correlations among the factors.

• (a) should follow from previous research, but (b) and (c) may well vary from one research context to another without undermining the usefulness of the questionnaire.

Self-report measures, ctd.• Exploratory factor analysis endeavours to find the most

meaningful factor structure in a set of data. • There are always three steps in these analyses. • First, you need to decide how many factors to identify in

a set of data. • Second, you need to decide which statistical model to

use to extract the factors from the data.

Self-report measures, ctd.• Third, you usually transform (or rotate) the factors in

order to obtain a more interpretable representation of the extracted factors.

• You need to decide whether to use an orthogonal rotation (the factors remain independent of each other) or an oblique rotation (the factors may be correlated).

• An orthogonal rotation is conceptually simpler. But who said life was simple? In any case, oblique rotation includes the possibility that the factors are independent.

Self-report measures, ctd.• Richardson (2006) carried out an exploratory factor

analysis on the scores obtained by students taking technology courses on the various subscales of the Revised Approaches to Studying Inventory.

• Preliminary tests suggested that there were three factors. These were extracted using principal axis factoring and submitted to an oblique rotation.

Self-report measures, ctd.• The loadings (i.e. correlations) of the first factor were

mainly on the subscales measuring a deep approach, those of the second mainly on the subscales measuring a strategic approach and those of the third mainly on the subscales measuring a surface approach.

• This supports the construct validity of the RASI in the novel research context of students taking courses in technology by distance learning.

Self-report measures, ctd. • The criterion validity of an instrument is the extent to

which the scores are correlated with some independent criterion or measure of the quality being measured.

• The concurrent validity of an instrument is the extent to which the scores are correlated with a criterion that is measured at the same time.

• The predictive validity of an instrument is the extent to which the scores are correlated with a criterion that is measured at some subsequent point.

Self-report measures, ctd.• The National Student Survey is an annual survey of all

final-year students in England, Wales and Northern Ireland. It contains 21 items in scales that reflect various aspects of the quality of the student experience.

• There is a 22nd item, “Overall, I am satisfied with the quality of the course.” This is not part of the NSS itself but is used to validate the scores on the NSS scales by showing that they are correlated with rated satisfaction (see Richardson, Slater, and Wilson, 2007).

Self-report measures, ctd. • Both forms of criterion validity can be investigated by

examining simple correlation coefficients between the questionnaire scores and the criterion.

• However, in a research context, it might be useful to use multiple regression analysis to investigate how important different scores are in predicting the criterion.

• For instance, some scales may only predict the criterion because they are confounded with other scales.

Self-report measures, ctd.• Students’ marks or grades are positively correlated with

their scores on deep approach but negatively correlated with their scores on surface approach.

• But their scores on deep approach may be negatively correlated with their scores on surface approach.

• In a multiple regression analysis, only surface approach may turn out to be a significant predictor of their marks.

• So the marking punishes the use of a surface approach but doesn’t directly reward the use of a deep approach.

Self-report measures• The discriminant validity of an instrument is the extent to

which it yields different scores on groups of people who would be expected to differ in the underlying traits.

• This can be studied by the use of multivariate analysis of variance, which tests whether two or more groups differ in their patterns of scores on two or more variables.

• If the overall comparison is statistically significant, it needs to be followed up by univariate analyses to find the particular variables responsible for the differences.

Self-report measures, ctd.• Richardson (2006) used multivariate analysis of variance

to compare the scores on the RASI obtained by students taking different technology courses.

• There were significant differences both across different courses and within the same course depending on when the RASI had been administered (middle versus end).

• Univariate analyses showed (for instance) that students’ understanding increased towards the end of the course.

Self-report measures, ctd.• Cluster analysis can be used to identify subgroups who

obtain different patterns of scores on a questionnaire. • For instance, Richardson (2007) used a questionnaire to

explore distance learning students’ conceptions or mental models of learning.

• A factor analysis identified five scales linking learning with the construction of knowledge, the intake of knowledge, the use of knowledge, stimulating education, or cooperative learning.

Self-report measures, ctd.• Cluster analysis can be used to identify subgroups who

obtain different patterns of scores on a questionnaire. • For instance, Richardson (2007) used a questionnaire to

explore distance learning students’ conceptions or mental models of learning.

• A factor analysis identified five scales linking learning with the construction of knowledge, the intake of knowledge, the use of knowledge, stimulating education, or cooperative learning.

Self-report measures, ctd.• A cluster analysis then identified four subgroups of

students with different patterns of scale scores. • A multivariate analysis of variance showed that students

in the four clusters had different scale scores.• Univariate analyses showed that the students in the four

clusters had different mean scores on all five scales.

Self-report measures, ctd.• This is not surprising, because the cluster analysis was

used to find subgroups of students whose mean scores were different from one another.

• However, the scores on the different scales were correlated and therefore confounded with one another.

• So the univariate analyses only describe the differences among the subgroups; they don’t identify which scales actually define membership of the different subgroups.

Self-report measures, ctd. • Discriminant analysis is used to identify which of a set of

variables predicts membership of two or more groups. • Richardson used a discriminant analysis to identify

which of the five scales predicted membership of the four clusters.

• One scale did not significantly predict membership of the four clusters when scores on the other four scales were taken into account.

Self-report measures, ctd. • But membership of the four clusters was predicted by

various combinations of scores on the other four scales. • The different patterns of scores were then used to

characterise four different conceptions or mental models of learning that are shown by distance learning students.

• In short, this study used factor analysis to reduce the data to a few scales, cluster analysis to find subgroups within the sample, and discriminant analysis to see which scales predicted membership of the subgroups.

Measures of effect size• Nowadays, most researchers appreciate that whether or

not an effect is statistically significant does not tell you whether it is of any theoretical or practical importance.

• The latter question can be addressed by calculating and reporting measures of effect size.

• These fall into two main types: measures of difference and measures of explained variation (Richardson, 1996).

Measures of effect size, ctd.Suppose we administer a technique aimed at improving performance in a class of students and find that their average score is 70 points.

For comparison, we could examine another class who had not received our technique, and we might find that their average score was 65 points.

Is this evidence of a significant improvement?

Measures of effect size, ctd.If the standard deviation of each group is just 1 point, then the difference between the two groups is five times that figure, which might be regarded as impressive.

However, if the standard deviation of each group is 20 points, then the difference between the two groups is just a quarter of that figure, which looks far less impressive.

Measures of effect size, ctd.In general, we can express the size of an effect by dividing the difference between the two means by the within-group standard deviation to give a standardised mean difference. An effect size of 1.0 implies that the mean score of one group is a standard deviation higher than the mean of the other group.

In general, the effect size d = (m1 – m2)/s.

In the first example, d = (m1 – m2)/s = (70 – 65)/1 = 5.00.

In the other example, d = (m1 – m2)/s = (70 – 65)/20 = 0.25.

Measures of effect size, ctd.This is a useful notion because it separates the magnitude of an effect from whether or not it is statistically significant.

It is also useful because it provides a way of expressing the magnitude of an effect in terms that are independent of the particular instruments and procedures that were used to measure it.

This means that effect sizes can be (a) compared across studies using different instruments and procedures and (b) combined using techniques known as meta-analysis.

Measures of effect size, ctd. • The standardised mean difference can only be used

when comparing two different groups of participants.• In other research designs, other measures of effect size

have to be used. • These typically seek to quantify the proportion of

variation in the dependent variable that is explained by different predictor variables.

• Statistics of this sort can be readily calculated using statistical packages (see Richardson, in press).

Conclusions• Think carefully and critically about the origins and

measurement properties of your data. • Be sure that you have a clear research problem in

advance of collecting data. • Think creatively about what kinds of statistical analysis

might be used to address your research problem.• Expect to provide measures of effect size as well as

more conventional statistical results.

ReferencesCronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.Entwistle, N., Tait, H. & McCune, V. (2000) Patterns of response to an approaches to studying inventory

across contrasting groups and contexts. European Journal of Psychology of Education, 15, 33–48. Richardson, J. T. E. (1994). The analysis of 2 × 1 and 2 × 2 contingency tables: An historical review.

Statistical Methods in Medical Research, 3, 107–133.Richardson, J. T. E. (1996). Measures of effect size. Behavior Research Methods, Instruments, and

Computers, 28, 12–22.Richardson, J. T. E. (2006). Perceptions of academic quality and approaches to studying among technology

students in distance education. European Journal of Engineering Education, 31, 421–433.Richardson, J. T. E. (2007). Mental models of learning in distance education. British Journal of Educational

Psychology, 77, 253–270.Richardson, J. T. E. (2008). The attainment of ethnic minority students in UK higher education. Studies in

Higher Education, 33, 33–48.Richardson, J. T. E. (in press). Eta squared and partial eta squared as measures of effect size in educational

research. Educational Research Review. Richardson, J. T. E., Slater, J. B., & Wilson, J. (2007). The National Student Survey: development, findings

and implications. Studies in Higher Education, 32, 557–580.Richardson, J. T. E. and Wydell, T. N. (2003). The representation and attainment of students with dyslexia in

UK higher education. Reading and Writing, 16, 475–503.

Institute of Educational TechnologyThe Open UniversityWalton HallMilton KeynesMK7 6AA

www.open.ac.uk