IT in Education Symposium 2004 1

Designing and Developing Designing and Developing Research Projects and Research Projects and

Dissertation on IT in EducationDissertation on IT in Education

Alvin KwanDivision of Information & Technology Studies

Faculty of Education

University of Hong Kong

IT in Education Symposium 2004 2

The aimThe aim

The aim of this seminar is to share with the audience some common mistakes I found from the dissertations or projects my students, who were mostly first-time educational researchers, so that those mistakes can hopefully be avoided when the audience conduct their own research in the future.

IT in Education Symposium 2004 3

My assumptionsMy assumptions

The audience have no experience of conducting academic research, in particular, in the educational sector.

Given the time constraint, I would talk more on how not to do educational research as I believe this will help the audience to pick up the key ideas of doing research rightly faster.

IT in Education Symposium 2004 4

Topics of discussionTopics of discussion

Types of educational research Characteristics of experimental studies Typical research tools Some common mistakes in research design Some common problems in writing a dissertation

IT in Education Symposium 2004 5

Types of educational researchTypes of educational research

Confirmatory researchResearcher has generated a theoretical model that

requires to be tested (e.g., can educational games motivate students’ learning?)

Exploratory researchTo study a phenomenon so as to identify the main

variables of interest, their relationships and any potential causal linkages (e.g., what types of educational games motivate students’ learning most?)

IT in Education Symposium 2004 6

Another classificationAnother classification

HistoricalHeavily rely on the use of source documents

Case studiesNo intention to generalize inferences to wider

populations Longitudinal

Study of individual over time Survey Experimental (which is our focus of this seminar)

IT in Education Symposium 2004 7

Typical characteristics of Typical characteristics of experimental studiesexperimental studies Identification of independent variables (including

any treatment variables) and dependent variable Active manipulation of treatment variables by the

researcher, e.g., adopting a new teaching practice The use of random assignment of units, typically

students, to each treatment type Measure and analyze any differences between the

results, i.e., values of the dependent variable, associated with different treatments

IT in Education Symposium 2004 8

Caveat 1Caveat 1

Research data may suggest association, but not causal relationship, between or among variables.

IT in Education Symposium 2004 9

Caveat 2Caveat 2

Too many potential independent variables to choose from Student factors: age, sex, IT skill, learning ability,

accessibility to computer, and family background, etc.

Teacher factors: teaching styles, teaching methods, and IT skill, etc.

School factors: school policy, principal’s leadership, IT equipment availability and accessibility, etc.

IT in Education Symposium 2004 10

Caveat 3Caveat 3

It is difficult to set up a proper control for comparison purpose in educational research.A random sampling strategy, especially

when the sampling size is small, cannot warrant a good control.

Pre-test may be required to demonstrate that control and experimental groups are properly established.

IT in Education Symposium 2004 11

Caveat 4Caveat 4

Ensure that you only change one independent variable at a time in a selected treatment or experimental group.If you cannot ensure that only one

independent variable is changed in a treatment group due to some practical constraints, do discuss it and argue that the values of those “uncontrolled” independent variables are pretty much unchanged.

IT in Education Symposium 2004 12

Differences between research in Differences between research in natural science and educationnatural science and education Hard to set up control experiments for comparison

purpose in educational researchToo many potential independent variables which

are not controllable Experimental results may not be reproducible

Same experiments cannot be applied to the same subjects without affecting the validity of the research results.

IT in Education Symposium 2004 13

Typical research toolsTypical research tools

Survey Direct observation Interview Focus group

IT in Education Symposium 2004 14

A remark on research toolA remark on research tool

It is often difficult to distinguish between qualitative and quantitative studies at the level of research technique or data collection procedures.The distinction between qualitative and

quantitative studies lies on how collected data are analyzed.

IT in Education Symposium 2004 15

Some common mistakes in Some common mistakes in research designresearch design Unclear or non-measurable research goals

Setting over-ambitious, vaguely defined or too many (sometimes even unrelated) research goals

Depth is more important than breadthFail to pose significant questions that can be

investigated empirically Examples of significant questions

Questions that address unexploited research areas Refining or refuting a prior relevant theory

IT in Education Symposium 2004 16

Caveat 5Caveat 5

Research goals must be measurable.Goals-> Perspectives -> IndicatorsE.g., if you want to study whether student

motivation can be improved after adopting ICT, you have to ask yourself in which aspects of learning that student motivation would exhibit and how signs of student motivation can be measured in each of those aspects.

IT in Education Symposium 2004 17

Some common mistakes in Some common mistakes in research design (continued)research design (continued) Doing your own research

Be aware of relevant prior research, if any, by doing a proper literature review so that

you won’t reinvent the wheel you would have a better idea of how to justify the

value of your research among other similar research you can discuss your work in light of others,

especially relevant theory

IT in Education Symposium 2004 18

Caveat 6Caveat 6

Arguments must be substantiated.If you cannot “prove” it (from the data you

collected), then “steal” it (from other research).

The legitimate way of “stealing” ideas is referred to as “proper citation”.

IT in Education Symposium 2004 19

Some common mistakes in Some common mistakes in research design (continued)research design (continued) Interpreting statistics in your own ways

Beware of statistical errorBe aware of any assumptions of a statistical test,

e.g., normality of data for parametric statisticsComputing means of ordinal data can be

misleading

IT in Education Symposium 2004 20

Some common problems in Some common problems in writing a dissertationwriting a dissertation Too eager to tell the audience all you know

Convey a coherent message in your dissertationUse appendix if necessary

Too “close” to your workLogical gap in arguments could be difficult to detect

by yourselfDistance “yourself” from your work before re-

reading

IT in Education Symposium 2004 21

Some common problems in Some common problems in writing a dissertation (continued)writing a dissertation (continued) Telling your research results only

Are your research results in line with other similar research?

Too little time for the write-up processTeachers assess students’ dissertations largely based

on their dissertation.A poorly written dissertation can be “lethal”.

IT in Education Symposium 2004 22

Caveat 7Caveat 7

Negative results can be as meaningful as positive results.Provided that there is nothing wrong with

the research design, negative results often reflect possible inadequacies in current theories and may give hints at future research direction.

IT in Education Symposium 2004 23

Thank You!