hope research methods collecting, processing and analyzing data

HO

PE Research Methods

Collecting, Processing and Analyzing Data

Research Methods Tony Fleet 2

HO

PE

Aims of the Session

The purpose of this session is to:• Alert you to the different types of

methodology available to you in your research

• Make you aware of the different techniques that you might use in collecting, presenting and analysing data

• Discuss the different kinds of problems that you might encounter in pursuing your research.


HO

PE

Contents1. Developing Your Research Questions

2. The Different Types of Research

3. Selecting Appropriate Research Methods

4. Robustness of Methods

5. Structuring Your Methodology

6. Data Analysis

7. Problems with The Research Process

8. Summary

HO

PE

1. Developing your Research Questions

This section of the presentation examines what you need to do in order

to focus your research.


HO

PE

The Purpose of Research

The purpose of research is to contribute to a current academic debate, and possibly to advance knowledge in some manner. This means that the research that you undertake has to be :

1. Embedded in a recognisable field of study, taking account of, and drawing on, past research

2. Of interest to other researchers working in the same field, and possibly to the wider community

3. Generalisable to more than one individual experience or circumstance.


HO

PE

Typical Research Structure

Conduct Literature Review

Select Research Questions

Devise Methodology & Research Instruments

Apply Methods & Instruments

Perform Statistical Analysis

Test Hypotheses & Draw Conclusions

The process of research is well-documented. This diagram more or less describes the activities you need to undertake.

What we will do in this session, is to look at some of the elements, and how they fit together.


HO

PE

Getting StartedConduct Literature Review






The first thing that you will do is to make sure you are well-informed, and to pose pertinent questions which your research will answer.


HO

PE

Defining Your Field of Study (1)

• You undertake a Literature Review in a particular field, in order to ensure that your research is embedded within that field, and that you are taking account of the methods, issues, results, theories and conventions which apply.

• At the end of the literature review, you will have narrowed the field down to a relatively small topic within the field, and will have some ‘unanswered questions’ which your research will try to investigate.

• These focussed questions form the basis of your Research.


HO

PE

Defining Your Field of Study (2)Your Research Questions are crucial, as they effectively define both the content of your research and the manner in which you carry it out. Your research methodology is designed specifically to

attempt to answer these questions The data that you collect will be focussed on issues

relevant to these questions Your analysis of the data will seek to provide answers

to the questions Your conclusions will summarise the answers.


HO

PE

Research Questions

Typical research questions might be:– To what extent is the business community in

India aware of the potential of Bluetooth?

– Is the software currently available for teaching arithmetic to 5 year olds appropriate and effective?

– Are there differences in the way that men and women approach the task of writing software?

– How can historical events be modelled effectively using VRML?


HO

PE

Defining Your Population• When framing your Research questions, you need to be

clear about what set of objects, people or events forms the ‘background population’ in your study.

• Are you saying, for example that the CAL software you produce is designed for all English-Speaking people, all men, all Afro-Caribbeans, all children under 5, all those who have been diagnosed as dyslexic, or simply to Afro-Caribbean boys under 5 with specific learning difficulties?

• If you are investigating whether on-line learning is effective, is your population: students, University students, UK University students, Liverpool Hope Students, Liverpool Hope Computing Students or Liverpool Hope MSc. Computing Students?


HO

PE

Research QuestionsReferences (SW Library)

• Lewis, Ian. - So you want to do research! : a guide for beginners on how to formulate research questions. - 2nd ed. - Edinburgh : Scottish Council for Research in Education, 1997. - (SCRE publication ; 2 ; .. - 1860030327

HO

PE

2. The Different Types of Research

Here we look at the different options available to us in carrying

out the research.


HO

PE

Your Research Focus

The main focus of your research can be:• Product-based Research, focusing on producing a piece

of hardware or software (or the designs for them) which is at the ‘cutting edge of a discipline, drawing on other researchers’ ideas, best practice and what is feasible. In doing this, you may need to explore how the product will ‘enmesh’ with current systems, and existing and future technologies.

• People-based research, focusing on the people who interact with the hardware or software, looking at issues such as usability, user behaviour, compatibility of software with current user systems and other HCI issues.Both of these approaches are legitimate, and it is possible that in carrying out your research you might need to use elements of each one.


HO

PE

Product v. PeopleWhich focus?

• The focus of your research is decided by you. • It will depend upon how confident you are in creating a

product at the ‘cutting edge’, or how comfortable you will be as a researcher in dealing with people.

• When you frame your research questions, you need to ensure that their focus leads you into the kind of research that you want to do.

• There is no right answer, but you may find that your research will be best carried out by using a ‘main’ focus of one element, with a subsidiary focus of another. For example producing a piece of software which securely encodes personal data as a self-encrypting and decrypting file stored on an ID card, may well need trialling with real people.


HO

PE

Approaches to Research

There are two main approaches to doing research:• Quantitative Research looks for ‘hard’ numerical

data; it proceeds by counting, measuring and summarising. The goal is to look for statistical significance in the results.

• Qualitative Research takes a ‘soft’ approach to data, and has a more descriptive ‘feel’. It attempts to get to the heart of the matter, exploring individual cases in detail, and seeking the reasons for behaviour and explanations for events.

Both of these approaches are legitimate, and it ispossible to combine elements.


HO

PE

Quantitative v. QualitativeWhich approach?

• The approach you use will depend upon your topic and your research questions.

• It will also depend upon how comfortable you as a researcher feel about using these methods.

• There is no right answer here, and, as we shall see in the rest of this presentation, there may be good reasons for adopting a variety of methods, which encompass both quantitative and qualitative approaches


HO

PE

Types of Research

There are four main types of research that you might consider: Experimental Research Survey Research Evaluative Research Observational Research

All four of these types can incorporate both quantitative and qualitative approaches


HO

PE

Experimental Research

This is normally quantitative, but can take two forms:• An attempt to produce a piece of hardware,

software or a combination of both, which is at the ‘cutting edge’ of a discipline.

• An attempt to investigate and document the performance of a particular piece technology in specific circumstances.

This might involve:• Creating hardware or software applications• Devising detailed tests and evaluation procedures• Carrying out rigorous testing• Evaluating performance or usability


HO

PE

Survey Research

This research can be qualitative or quantitative; in the widest sense, you are interviewing people. This might involve:

• An unstructured interview• A semi-structured interview• An structured interview based on questionnaire (face

to face, or by telephone)• An administered questionnaire• A Postal Questionnaire


HO

PE

Evaluative Research

This is primarily qualitative. Here you are trying to assess whether something is of value, whether it meets its specifications, or whether it is fit-for-purpose. This might involve:

• Developing a list of criteria on which to make judgements• Examining the object against each of the criteria to judge

to what extent it conforms to expectations• Weighing the positives and the negatives, coming to

overall conclusions• Matching these judgments against similar judgements

made elsewhere in the literature or in real life.


HO

PE

Observational Research

This research normally uses a qualitative approach; in the widest sense, you are recording people’s behaviour. This might involve:

• Participating in a task or situation• Making field notes of experiences• Creating and using an Observation Schedule• Making a check-list of occurrences of particular

events or items.


HO

PE

Research possibilities

Qualitative Quantitative

Experiment Field- testing Bench-testing and simulations

Survey Unstructured Interviews

Written Questionnaires

Evaluation Using expert judges Evaluation criteria and checklists.

Observation Participant Observation

Observation schedules


HO

PE

Other Forms of Research

Historical & Documentary Research proceeds by scrutinising existing materials, both written and artefacts, using them as sources of evidence.

Action Research is normally conducted in an educational or political context. Action is taken, monitored, evaluated and then modified for the next cycle.

Ethnographic Research consists of an in-depth study of a cultural phenomenon, in order to generate new theory.

Case Study Research selects a whole range of research methods in scrutinising one particular context or situation.


HO

PE

Research methods 1References (SW Library)

• Crabtree, Benjamin F. - Doing qualitative research. - London : Sage, 1992. - (Research Methods for Primary Care ; 3). - 0803943121

• Creswell, John W.. - Research design : qualitative and quantitative approaches / John W. Creswell. - Thousand Oaks, Calif; London : Sage, 1994. - 0803952554

• Creswell, John W.. - Qualitative inquiry and research design : choosing among five traditions / J. - Thousand Oaks, Calif.; London : SAGE, 1998. - 0761901434


HO

PE

Research methods 2References (SW Library)

• Miller, Delbert Charles. - Handbook of research design and social measurement. - 3rd ed. - New York : David McKay Co . Inc, 1977. - m0859739

• Research methods in education and the social sciences / [Research Methods in. - Block 3B : Research design. - Milton Keynes : Open University Press, 1983. - (DE304, Block 3B ; 3B). - 0335074235

• Yin, Robert K.. - Case study research : design and methods. - Rev. ed. - Newbury Park; London : Sage, 1989. - (Applied social research methods series ; v.5). - 080393470x

HO

PE

3. Selecting Appropriate Research Methods

The next few slides discuss how you might go about selecting your

research methods from those available


HO

PE

Selecting Your Methodology

Your research methodology consists of :• Research Methods (experiment, survey etc.)• Research Instruments (questionnaire, tests etc.)• Analytical Tools (statistics, inductive or deductive

methods)

When selecting the methodology, you need to be aware of:

• The Research Questions you are trying to answer• The Population you are trying to generalise to.


HO

PE

Factors to Consider

Undertake Literature Review


How has previous research in this area been

done?




What methods have been used?

What research instruments have been devised?

How will the methods & instruments be applied?

What Statistical tests can be carried out?


What Research Hypotheses can be tested?


HO

PE

Appropriate methodology

• Do your Research Questions involve impressions, attitudes, opinions, beliefs or knowledge held by people?

• If so, then survey research is appropriate

• Do your Research Questions involve behaviour, actions, reactions to events, circumstances or objects?

• If so, then observational research is appropriate


HO

PE

Appropriate methodology

• Do your Research Questions involve the reliability or robustness of hardware, software, systems or infrastructure?

• If so, then an evaluative study is appropriate

• Do your Research Questions involve the testing of hardware or software at the technical level, speed, accuracy, security etc?

• If so, then experimentation is appropriate


HO

PE

A Mix of Methods

• It may be that your research questions overlap some of these categories, or different questions address more than category.

• If so, you should consider a mix of methods, that ensure that you cover all eventualities. This may bring added benefits. (See Triangulation)


HO

PE

Mixed MethodsReferences (SW Library)

• Mixing methods : qualitative and quantitative research / edited by Julia Bra. - Aldershot : Avebury, 1995. - 1859721168

• Tashakkori, Abbas. - Mixed methodology : combining qualitative and quantitative approaches / Abba. - Thousand Oaks, Calif.; London : Sage, 1998. - (Applied social research methods series ; v.46). - 0761900705

HO

PE

4. Robustness of Methods

As well as ensuring that your questions are well-focussed, and your methods relevant and appropriate, you need to

ensure that your methods are also Reliable and Valid


HO

PE

Reliability & Validity

• Research is Reliable if different methods, researchers and sample groups would have produced the same results.

• Research is Valid if the results produced by the research are accurate portrayals of the situation, explanations are effective, and predictions from the research are actually borne out by observation.


HO

PE

Reliability

• Research can have poor reliability, if it on one or two cases only, or if personal judgement or opinion is included.

• Reliability can be improved if data collection methods are made more precise, we have controlled experimentation, and we can produce statistical summaries.


HO

PE

Validity

• Research can have poor validity if the data produced is too far removed the object under study, or the respondent. Using detailed, highly structured research instruments can lead to distortions, by forcing observations into categories when they do not fit. Data summaries and averaging can also lead to distortions, and meaningless generalities. Graphical representations and percentages can be highly selective and produce biased findings.

• Validity can be improved by working directly with individuals or objects, focusing on specific cases, making detailed observations, conducting face to face interviews, taking detailed measurements in specific circumstances etc.


HO

PE

Reliability v. Validity

Valid, but not reliable


HO

PE

Reliability v. Validity

Reliable, but not Valid


HO

PE

Methodological Trade-Off

• If you improve validity, you will almost certainly reduce reliability.

• If you improve reliability it will be at the cost of reducing your validity.

• The trade-off is to balance the two so that the benefits of using particular methods outweigh the losses incurred.


HO

PE

Triangulation

• Triangulation takes its name from the navigational method of positioning a ship at sea by making two independent observations.

• The purpose here is to use two distinct methodologies, independent of one another to confirm that the effects which we are observing are real, and not artefacts of the research process.


HO

PE

Triangulation

• Triangulation attempts to counter the methodological trade-off by using a mix of methodologies.

• If you are using highly structured, statistical or measurement-based research, you supplement this with detailed observations or face to face interviewing.

• If your research is mainly based on individuals or on single items, you ensure that at least part of it has some statistical summaries, structured observations or questionnaires.


HO

PE

Validity & ReliabilityReferences (SW Library)

• Kirk, Jerome. - Reliability & validity in qualitative research. - Beverly Hills Calif : Sage Pubns, 1986. - (Qualitative Research Methods Series ; 1). - 0803924704

• Litwin, Mark S. - How to measure survey reliability and validity. - London : Sage, 1995. - (The Survey Kit ; 7). - 0803957041

HO

PE

5. Structuring Your Methodology

This section looks in detail at the techniques that you might

employ in the Research Activity itself.


HO

PE

Research Structure







After framing your Research Questions, and selecting your methodology, you should test that this is going to work by conducting a small-scale Pilot Study


HO

PE

5a Pilot Studies

• A pilot study is a set of preliminary investigations & procedures carried out prior to the main research, to ensure that the research is possible and can proceed without hitches.


HO

PE

Pilot Study (1)• In almost every type of Research, you will

need to devise or adapt some sort of Research Method or Instrument.

• This might be a measuring procedure, a set of evaluation criteria, an interview procedure or questionnaire, or an observational method or schedule.


HO

PE

Pilot Study (2)• Your procedure or instrument should be

based on best practice from previous research.

• It is unlikely that you will find exactly what you need; you will be forced to adapt or amend it.

• This means that you will need to conduct a Pilot Study to test whether the new instrument is fit-for-purpose.


HO

PE

Pilot Study (3)• The main purpose of a Pilot Study is to iron

out ‘bugs’ in procedures, or to check that instruments work.

• The size of the pilot study will depend on how ‘inventive’ you needed to be.

• If your procedures are almost entirely of your own devising, then you will need a fairly extensive pilot study to check them.

• If you have ‘lifted’ methods from the literature, than your pilot can be quite small.


HO

PE

Pilot Study (3)

• With questionnaires & observation schedules, you will need to check that individual items are giving you expected results.

• With test procedures, you need to check that you can actually do what you have said that you are going to do.

• With evaluation criteria, you need to use these in a limited context, to see that they are workable and effective.


HO

PE

Pilot Study (4)• As a result of the Pilot Study, you need to

evaluate procedures and instruments, making amendments where necessary.

• The Pilot Study stage will be part of your research; you will need to write this up, reporting on how your methods were adapted and improved as a result.


HO

PE

Carrying Out the Research







Here we will examine how particular methods and instruments can be applied in the research situation.


HO

PE

5b A detailed look at some Research methods

• Experimental Design

• Evaluative Research

• Observational Research

• Survey Research


HO

PE

5c.1 Experimental Design

This section looks at the different ways in which you can conduct experiments. Note that you do not need to be doing pure ‘experimental’ research to adopt these methods.


HO

PE

Experimental Designs• You may need to think about experimental

design even if you are doing types of research other than ‘Experiments’

• These designs occur where you have made some change, and are trying to find out its effect.

• The net result is that you are comparing one thing, or one group with another thing or group.


HO

PE

Experimental Designs

• The terminology for experimental designs comes from agricultural experiments.

• We have different ‘treatments’ which we apply to different groups

• We control the groups for different ‘factors’


HO

PE

Experimental Designs• Experiments normally involve ‘independent’ and

‘dependent’ variables’• Independent variables are factors that can be

controlled for, like temperature, file sizes, age, gender etc.

• Dependent variables are those factors which will change as a result of altering the independent variables.

• For example, download times will increase as a result of increasing file sizes. – Download time = dependent variable, – File size = independent variable


HO

PE

Pre-Test/Post TestControl Group Design

• This is the classic experimental design.

• It allows you to split your sample into two distinct parts (A & B)

• You give the same test to the two groups before you start

• You treat one of the groups• You apply the same test

afterwards.

Group A

(treated)

Group B(untreated)

Pre-test Pre-test

Treatment

Post-test Post-test


HO

PE

Control Group

• The Control Group (B), is one which is the same in all respects, except for the fact that we make no changes.

• We can use the test measurements on the control group as a benchmark for any changes we make to the treatment group (A)


HO

PE

Example 1: Server Testing• Suppose we wish to check whether a

firewall is effective in blocking external attacks.

• Set up System A & System B on two different servers, running ‘near-equivalent’ internal & external programs.

• Devise a test which includes a full range of possible attacks; apply to both A & B. Take series of measurements & observations.

• Incorporate firewall into System A• Reapply test to both systems, which are

again running ‘near-equivalent’ programs.

System A

(treated)

System B

(untreated)

Pre-test Pre-test

Apply Firewall

Post-test Post-test


HO

PE

Example 2: Software Design • Suppose we wish to find out whether

anthropomorphic agents are useful in communicating with novice users of a website.

• Set up Website A & Website B with ‘near-equivalent’ structure, but different content.

• Devise a procedure which asks two equivalent sets of users to make explorations of the websites; apply to both A & B. Make a series of observations

• Incorporate Agents into Website A• Re-apply procedure, asking two more

equivalent sets of users to make the same explorations of both Websites. Again make observations.

Website A

(treated)

Website B

(untreated)

Pre-test Pre-test

Apply Agents

Post-test Post-test


HO

PE

Example 3: On-Line Learning• Suppose we wish to find out whether training is

effective in helping students cope with the demands of on-line learning.

• Set up group A & group B with ‘near-equivalent’ members.

• The two groups to undertake a short programme of learning on-line, which incorporates a short evaluation and a knowledge test.

• Give Training to Group A• The two groups to undertake a further short

programme of learning on-line, again incorporating a short evaluation and knowledge test.

Group A

(treated)

Group B

(untreated)

Pre-test Pre-test

Apply Training

Post-test Post-test


HO

PE

Factorial Designs• This is where we examine the effects of two or

more independent factors simultaneously.• For two factors, we would need 4 groups

– Group A (Control: no treatment)– Group B (Factor 1 treatment only)– Group C (Factor 2 Treatment only)– Group D (Factors 1 & 2 Treatments)

• Clearly, this is going to increase the complexity and size of the research, but it has the added benefit of producing verifiable results in cases where two factors interact to produce interesting effects.


HO

PE

Other Design VariantsPost-Test only Control Group Design• Here we assume that both groups are the same

(but do not test that assumption). We simply apply the treatment to one group, and apply the test to both groups.

Matched Pairs Design• The individuals we use in the test groups

(subjects) are matched for characteristics which are likely to affect the outcome (gender, age, level of education, ethnicity etc.)


HO

PE

Experimental Designs:Reliability Issues

• For high reliability, we need to build in strict controls over each of the independent variables affecting the outcomes of the experiment.

• We will also need to ensure that any measurements taken of the dependent variables are as accurate as possible.

• We also need to ensure that our methodology is clear and replicable.


HO

PE

Experimental Designs:Validity Issues

• For high validity, we need to conduct the experiment in as ‘natural’ a setting as possible, and in as near as exact a match to the circumstances in which the events or objects would normally operate.

• If a sample is being used, then the sample (whether time, events, people or objects) should be as representative of the ‘background’ population as possible, and we should make detailed observations of how the events unfold as well as the final measurements.


HO

PE

Experimental DesignReferences (SW Library)

• Campbell, Donald T., Donald Thomas, 1916-. - Experimental and quasi-experimental designs for research / Donald T. Campbell. - Boston; London : Houghton Mifflin, 1963. – 0395307872

• Field, Andy. - How to design and report experiments / Andy Field, Graham Hole. - London : SAGE, 2003. - 0761973826

• Miller, Steve. - Experimental design and statistics. - London : Methuen, 1975. - (Essential Psychology ; A8). - m0805407


HO

PE

5c.2 Evaluative Research

This section looks at the methods and issues surrounding evaluation. You may need to use such techniques if evaluation is implicit in your research questions


HO

PE

Evaluative Research• Evaluative Research covers those cases where

you are attempting to compare whether one procedure or object is better or more effective than another procedure or object, or to determine whether a particular procedure or object is fit-for-purpose.

• Evaluation needs to be done against a set of criteria, which have been established as valid and reliable in this context. You would normally produce an evaluation form to be completed by respondents.


HO

PE

Evaluation Criteria• To establish criteria for evaluation, we need to break the

topic down into individual elements, and state the different sub-topics on which the item is to be evaluated.

• Alongside this, we will normally state specific questions which will need to be answered in order to judge the item against the criterion.

• The answers to the questions may involve measurements, counts, assessments on subjective scales, statements of fact or possibly even opinion.


HO

PE

Example: Website Evaluation Criteria

Design • Is the use of colour acceptable? • Are the elements in harmony?Navigation• Do the links work? • How many links per page?Content• How many words & graphics on page? • Is the text interesting & informative?Interactivity• What interactive features are used?• Do these improve communication with the user?Coding• What scripting is used? Is it clearly annotated?


HO

PE

Example: Website Evaluation Form

1. Design • Colour Safe Pallet used? Yes/ No• Harmony of elements on page: good/ neutral/ poor2. Navigation• Number of links on the page: ____• Number of working links ____3. Content• Total size of graphics files on page: ____ MB • Is the text: boring/ dull/ neutral/ interesting/ fascinating4. Interactivity• Tick and/or name all interactive features used:

rollovers; dynamic images; image maps; ______, ______ 5. Coding• What DTD has been used? _______________________


HO

PE

Evaluation Criteria:Validity Issues

• For criteria to be valid, each of the criteria elements need to have face or content validity; the questions posed by the criterion needs to be relevant to the object, and relates to a feature of it

• We also need to be able to answer the questions in an objective manner, without recourse to guessing, or giving an ‘impressionistic’ response.


HO

PE

Evaluation Criteria:Reliability Issues

• For criteria to be reliable, each of the criteria elements need to be clear and unambiguous, so that different assessors would interpret the criteria in the same way.

• We also need to ensure that repetition of the evaluation exercise will yield results which are not dissimilar to one another.


HO

PE

Evaluation MethodsReferences (SW Library)

• Britain, Sandy. - A framework for pedagogical evaluation of virtual learning environments. - Manchester : Joint Information Systems Committee, 1999. - (JISC Technology Applications Programme r.. - M0000712EL

• Broadbent, George Ernest. - The role of evaluation in user interface design : a practical implementation. - Liverpool : University of Liverpool, 1997. - p7270369

• Redmond-Pyle, David. - Graphical user interface design and evaluation (GUIDE) : a practical process. - London : Prentice Hall, 1995. - 013315193x

• Smeltzer, Nicholas. - Critical analysis of the design and evaluation of a computer-based project. - Liverpool : University of Liverpool, 2001. - M0002704LO


HO

PE

5c.3 Observational Research

This section describes what you need to do if your research involves making detailed observations of people, events or objects.


HO

PE

Observational Research

In this case you are trying to determine

Either: How an individual or group of individuals react, interact or behave in particular circumstances

Or: How software or hardware performs when used by particular groups of people


HO

PE

Participant ObservationIn this case, the researcher becomes one of the subjects, and works alongside the subject, monitoring behaviour and interacting with them.

Pros:Researcher can fully ‘understand’ what the issues are. Can get real validity to research.

Cons: Researcher can influence the research, alter opinions. Highly subjective & unreliable. Taking notes is difficult; relies on good memory.


HO

PE

Non-Participant ObservationIn this case, the researcher studies the situation apart from subjects, taking notes, monitoring behaviour and observing interactions. The use of audio & video recording is useful here.

Pros:Researcher can get overview of the situation, and achieve objectivity

Cons: Researcher can only see resulting behaviours, not what is causing it or why it is happening.


HO

PE

Observation Schedules• An observation schedule can be a

simple list of things to look for in a particular situation

• It can be far more complex; a minute by minute count of events such a mouse-clicks or verbal interactions between subjects.


HO

PE

Observation ScheduleAn Example

Observation of subject using Information Portal

Subject: M / F Age: 18-21 / 21-30 / 31-50 / 50+Date: _________ Time: _________

Selected Navigation Tool : Mouse /Keyboard/ TouchScreenFirst 5 pages visited in order: __ __ __ __ __Time to obtain to obtain required information: ___ min ___ secTotal Number of Pages visited: __Feedback from subject:

very positive / positive / neutral / negative / very negative


HO

PE

Observation Schedules:Some Reliability Issues

• For an observation schedule to be reliable, it should require structured documentation of events.

• This will involve such things as checklists, minute-by-minute categorisation of activity and numerical data such as frequencies of occurrence and time intervals. Timings should have a clear start and end points.

• The schedule should leave little room for subjective judgement.


HO

PE

Observation Schedules:Some Validity Issues

• For an observation schedule to be valid, it should refer to events which actually happen; each of the events on the sheet should be possible, and likely to occur.

• Timings should be possible to take, and not interfere with other observations which should be made.

• There should be room for observations which ‘enrich’ the data by adding detail to the numbers, offering explanation and illumination.


HO

PE

Observational ResearchReferences (SW Library)

• Harding, Jacqueline. - How to make observations & assessments / Jackie Harding and Liz Meldon-Smith. - 2nd ed. - London : Hodder & Stoughton, 2000. - 034078038x

• Robertson, Kevin. - Observation, analysis and video / editors: Anne Simpkin and Penny Crisfield. - Leeds : National Coaching Foundation, 1999. - 1902523164

• Simpson, Mary. - Using observations in small-scale research : a beginner's guide / Mary Simpson. - Glasgow : Scottish Council for Research in Education, 1995. - (SCREpublications ; 16 ; 130). - 1860030122


HO

PE

5c.4 Survey-Type ResearchThis section describes what you need to do in order to use human respondents to provide you with information.


HO

PE

Survey methods

There are two distinct elements here:• The interview techniques that you adopt

in order to elicit information from people• The sampling methods that you adopt in

order to select respondents for interview

You will need to make rational choices for both of these elements, depending upon the focus of your research and the population under study.


HO

PE

Interviewing

• We interview people face-to-face in order to try to find out exactly what they think.

• With the right questions, people respond with high quality information.

• The data that you get can be of high validity, since you have access to respondents’ own words.


HO

PE

Types of Interviews• Unstructured Interview

• Structured Interview

• Semi-Structured Interview

• Administered Questionnaire


HO

PE

Open or Closed?

• When interviewing respondents, the main choice facing the researcher is whether to use open questions, which leave the respondent free to answer in any way they think fit, or closed questions which force the respondent to make particular choices, pre-determined by the researcher.

• OPEN: What is your experience of chat rooms?• CLOSED: Do you think chat rooms should be

monitored? (Yes/No/Maybe)


HO

PE

Open or Closed?Validity & Reliability Issues

• In general, the data from open questions is richer, more illuminating, and more valid, as it can illustrate clearly why particular subjects think the way that they do, or why they behave in particular ways.

• Data from closed questions however, is more amenable to statistics. It is easier to summarise, spot trend and make comparisons. In general the data is more reliable.

• A good strategy is to use a mixture of both open and closed questions


HO

PE

Unstructured Interviews

Unstructured Interview• Interviewer has no set agenda; the object is to get

the respondent to talk freely about various topics.Pros: Can be high quality data, often get real

insights into what people really think.Cons: Can be difficult for the novice researcher to

carry out; need to be good at steering the conversation without forcing it. Time consuming; can only carry out a small number. Difficult to analyse


HO

PE

Semi- Structured Interview

• Interviewer has a formal list of topics, which sets the agenda; however, the interviewer is free to take these in different orders, or to return to topics at different points.

• Here we include the idea of ‘focus’ groups, where a facilitator encourages the discussion of a particular topic.

Pros: Data provided can be almost as good as unstructured interview, but in a more focussed manner.

Cons: Can miss important ideas, because agenda set beforehand; interviews can also be time consuming & difficult to manage & analyse afterwards.


HO

PE

Structured Interviews

• Interviewer has a list of topics to be taken in a particular order; questions are written, and read out.

Pros: You can make good comparisons between different respondents; you also will have access to respondents’ thinking.

Cons: Unless you have done some preliminary investigations, and extensively piloted the interviews, you may miss lots of important data. Takes time to do properly; can only carry out a few.


HO

PE

Administered Questionnaire

• Interviewer has devised a list of questions which are read out; some multiple choice, some open-ended. Filled in either by respondent or interviewer.

• Pros: Interview can be quite brief; easy to make comparisons, data amenable to statistical analysis.

• Cons: Data may be ‘warped’ by the choice of questions. Respondent may be forced into giving fabricated answers.


HO

PE

QuestionnairesThere are many different types of questionnaire, depending upon what you are trying to find out.As well as gathering factual data about the person, you may be trying to explore their:– Knowledge – Beliefs – Attitudes – Opinion– Behaviour

If you need to design a questionnaire, see for example: http://www.leeds.ac.uk/iss/documentation/top/top2/index.html


HO

PE

Question Types

• Factual Data• Binary (Yes/No)• Single Selection from Categories• Multiple Selection from Categories• Attitude Scale items• Focussed items to elicit categories• Conditional Questions for routing• Open-ended Questions• Self-Reporting


HO

PE

Writing Questions• Put Factual Data (demographics) at end• Put Explanations and Disclaimers at the start• Keep questions as brief as possible• Use non-technical language where possible• Avoid leading respondents towards particular answers• Use direct questions, no hypotheticals• Use simple questions, no portmanteaus• Use a mixture of positively and negatively worded

questions• Verify data by asking same question in different ways.

See: http://www.analytictech.com/mb313/principl.htm


HO

PE

Questionnaire Example1. Which operating system are you currently using:

Linux Windows Other

2. How would you rate the operating system?very poor poor good very good

3. Circle the tasks which you use your computer to do:Word processing Internet Access Program Development

4. Estimate the number of hours you use the computer for each week:

______ hours

5. Do you use broadband?Yes No


HO

PE

Protocol

Whatever your interviewing method, as part of the ethical constraints on Hope Researchers, you are required to do the following:Explain to the respondents what the research is

about.Tell them that they will not be identified by name

in the final report, and that any views that they express will be in confidence.

Explain to them that if there are questions with which they feel uncomfortable, they do not have to answer.


HO

PE

Attitude Measurement

• You may wish to incorporate ‘attitude measurement’ into your questionnaire as either a major or a minor feature.

• There are several different types of scales which can be used to elicit numerical measurements of attitude:– Likert Scaling– Thurstone Scaling– Guttman Scaling– Semantic Differential Scaling

See: http://www.socialresearchmethods.net/kb/scalgen.htm


HO

PE

Likert Scales (1932)

• This consists of items like “I would not trust a bank’s website to keep my details secure”

• Respondents are asked to use a scale such as 1=Strongly disagree; 2=disagree; 3=neutral; 4=agree; 5=strongly agree

• Scales can vary: 0-4, 1-7 etc.• Some proponents suggest removing the neutral

category to force a choice, but this can reduce validity.• Scores on individual items are totalled to obtain the

respondent’s score; this is often shown as an average.


HO

PE

Thurstone Scales (1928)

• This pre-ranks 11 statements with numerical values 1-11; each statement carries a numerical score, for example:• 1 = “Teleworking is an impossible concept”• 2 =“Teleworking may be OK in exceptional cases”• 8 = “If offered the opportunity, I would try teleworking”• 11 = “In 20 years’ time, everyone will be teleworking”

• Clearly, lots of preparatory work needs to be done to generate and rank the statements.

• A respondent’s score, is the average numerical value of all the items they agree with.


HO

PE

Guttman Scales (1944)

• This pre-ranks statements in order, very similar to Thurstone scaling, except here we attempt to construct the scale so that if a person agrees with item 4 on the scale, they will also agree with items 1,2 and 3.

• When the questions are administered, the items are muddled, but each retains a ranking.

• The respondent’s score is the sum of the ranks associated with the items he or she agreed with.


HO

PE

Osgood’s (1957)Semantic Differential Scales

• Respondents are asked to rate an idea or an object against a series of opposing adjectives or descriptions, for example:

Using LearnwiseExciting [ ] [ ] [ ] [ ] [ ] [ ] DullHard [ ] [ ] [ ] [ ] [ ] [ ] EasyFrustrating [ ] [ ] [ ] [ ] [ ] [ ] Stimulating

• The scale asks respondents to tick the box nearest the descriptor that they agree with. Each box has a numerical value: e.g. 1,2,3… 7

• The respondent’s score can be portrayed graphically, or as a total of numerical values.


HO

PE

Questionnaires & Interviews:Reliability Issues

• For questionnaire results to be reliable, you need closed questions that are precisely framed, unambiguous, with response categories that are well-defined, exhaustive and exclusive. You need to ask the same question in different ways, and you need to collate the information into statistical summaries and expose the data to rigorous statistical testing.

• You need to use as large a sample as possible, and even out any random fluctuations in the data by using summary statistics and hypothesis testing.


HO

PE

Questionnaires & Interviews:Validity Issues

• For questionnaire results to be valid, you need response categories which enable the respondent to express their views accurately; this can best be done with open questions. You should take time with the respondent and respect the data that they provide for you.

• You need to ensure that the sample is representative of the population, and large enough for the results to be statistically significant.


HO

PE

Questionnaire References On-Line

• http://www.tardis.ed.ac.uk/~kate/qmcweb/qcont.htm

• http://www.leeds.ac.uk/iss/documentation/top/top2.pdf

• http://www.statpac.com/surveys/• http://www.fao.org/docrep/W3241E/w3241

e05.htm#chapter%204:%20questionnaire%20design

• http://www.surveysystem.com/sdesign.htm


HO

PE

Questionnaire References (1)(SW Library)

• Foddy, William. - Constructing questions for interviews and questionnaires : theory and practice. - Cambridge : Cambridge University Press, 1993. - 0521467330

• Fowler, Floyd J.. - Improving survey questions : design and evaluation. - London : Sage, 1995. - 0803945833

• Frazer, Lorelle. - Questionnaire design and administration : a practical guide / Lorelle Frazer. - Brisbane : Wiley, 2000. - 0471342920

• Gillham, W. E. C., William Edwin Charles, 1936-. - Developing a questionnaire. - London : Continuum, 2000. - (Real world research). - 0826447953


HO

PE

Questionnaire References (2) (SW Library)

• Oppenheim, A. N., Abraham Naftali, 1924-. - Questionnaire design, interviewing and attitude measurement / A.N. Oppenheim. - New ed. - London : Continuum, 2000. - 0826451764

• Wengraf, Tom. - Qualitative research interviewing : biographic narrative and semi-structured interviews - London : SAGE, 2001. - 0803975007

• Young, Pauline V. - Scientific social surveys and research : an introduction to the background,. - Englewood Cliffs, N.J. : Prentice-Hall, 1966. - (Prentice-Hall Sociology Series). - m0859969

• Youngman, Michael Brendan. - Designing and analysing questionnaires. - Maidenhead, Berks : TRC Rediguide. - (Rediguide ; 12). - m0891542


HO

PE

Sampling

• The question of how to select the respondents for the sample is always tricky.

• The principle behind sampling, is that you should ensure that the sample is appropriate in order for you to generalise your results to the population under study.

• There are essentially three ways in which this is done:– Random Sampling– Quota Sampling– Stratified Random Sampling


HO

PE

Random Sampling

• This method requires you to get a list of all the population, as near complete as you can find, then use some random selection method (such as shutting your eyes and stabbing a pen at the list, or allocating using random numbers)

• The idea is that every person in the population has an equal chance of ending up in the sample.

• Most statistical methods assume that you are sampling randomly, and it is the only method which overall is guaranteed to ensure that samples are free from bias, and therefore provide validity.


HO

PE

Quota Sampling• Here, you identify particular sectors of the population,

such as men, women , those under 21, over 21, employed, unemployed etc., and put quotas on the number of people in each category.

• For example, in a quota sample of 12, we might have: – 2 males under 21; 2 females under 21– 4 males over 21 ; 4 females over 21

• The purpose here is not to get a sample which represents the population in its entirety, but to ensure that views from important sections of the population are represented.


HO

PE

Stratified Random Sampling• One of the problems with Random Sampling, is that if you take

small samples, you can very well end up with a biased sample , by chance.

• In Stratified Random sampling, you would measure what proportions of the population lie in each category or strata, and select your sample so that you get precisely those proportions in those strata

• For example in your population you might have: – 15% males under 21; 10% females under 21– 30% males over 21 ; 45% females over 21

• If you select a sample of 200 you would need stratified samples:– 30 males under 21; 20 females under 21– 60 males over 21 ; 90 females over 21

• The allocation of subjects to samples should be done randomly.• The purpose here is to get a valid sample which which represents

the population in its entirety.


HO

PE

Other Sampling MethodsSnowball Sampling: This method is used whenever you are

dealing with a tricky subject, where people may be doing devious or illegal activities (for example hacking, virus creation etc.). In this case, you would use one respondent to suggest the name of another who might be willing to be interviewed; that one would lead to several others and so on.

Cluster Sampling: This method might be used if you have a diverse population (such as those living in African urban communities). Here you would randomly select particular cities, and sample within neighbourhoods within those cities.

Convenience Sampling: This is used mainly for investigative research. It relies on the fact that people appearing at a particular location or time will do so, ‘at random’ (may not be true). We use this fact to stand in one place and interview them.


HO

PE

Sampling: On-Line References

This is an easy introduction:• http://www.csm.uwe.ac.uk/~pwhite/SURVEY1/

node26.html

This is more complex and technical:• http://www.socialresearchmethods.net/kb/external.htm

HO

PE 6. Data Analysis

This section of the presentation looks at the different types of data

available, and how this can be analysed


HO

PE

Analysis of Research DataUndertake Literature Review






Here we examine how the data produced by the research can be presented effectively and statistically analysed.


HO

PE

Data Analysis6a Types Of Data

6b Extracting Data for Analysis

6c Presentation of Data

6c Principles of Statistical Testing

6d Selected Tests

6e Analytical Tools


HO

PE

6a Types of Data

In this section we describe the different types of data that you might encounter, and what you might do with it in your study.


HO

PE

Types of Data

• ‘Measurement’-type data, given in units measured on a decimal scale. e.g. MB/sec, times, averages and other statistical summary values.

• ‘Counted’ data: number of occurrences of events or objects: e.g. number of pages visited

• ‘Ranking’ data: subjective impressions, such as marks out of 10, rankings etc. e.g. Interface rated as very poor (1), poor (2), good (3) or very good (4).

• Categorical data: Names of different things or categories, such as those named by an interviewee (NB these may be counted) e.g. “Manager A discussed 3 concerns: security, access and reliability”

• Descriptive data, such as that produced by interviews or observational field-notes: e.g. “Subject A tried to click on all the blue text, even though it was not underlined”.


HO

PE

‘Measured’ Data

• Such data is called “continuous”, “ratio scale”, or “real number” data

• This type of data is easiest to use; it can be arithmetically manipulated (added, subtracted, multiplied, divided, summarised)

• it is also likely to have good, predictable statistical features.

• There are lots of methods to pick from.• The inclusion of valid measured data will add

reliability to your study.


HO

PE

‘Counted’ Data

• Such data is normally called “discrete”, “interval scale”, or “integer” data

• It can be added, subtracted and averaged.• ‘Counted’ data, has almost all the features of

measured data, especially if the numbers of the count are large (above 30 or so).

• For small numbers (e.g. less than 30 observations in total), methods are limited.

• The inclusion of valid counted data will almost certainly improve the reliability of your study.


HO

PE

‘Ranking’ Data• Such data is normally called “interval scale data” data.• In order to use it effectively, there are several

techniques; the usual one is to convert the data into numerical values (e.g. “Yes” = 1, “No” = 0)

• When this has been done, it can be added, subtracted and averaged.

• ‘Interval’ data needs care; the data ‘looks like’ counts where the numbers are small, but are simply subjective impressions. To deal with such data we need to summarise large volumes of it data.

• The inclusion of valid counted data, suitably summarised can add reliability to your study.


HO

PE

‘Categorical’ Data

• Such data is normally called “nominal” data.• In order to use such data effectively, the objects, classes or ideas

that these categories represent should have the items within them counted, evaluated on an interval scale and/or summarised.

• A preliminary study may have found, for example that there are 4 main uses for spreadsheets in business: Finance, Timesheets, Inventory and General Calculation. Businesses may be asked to rank these in order of importance, or to rate their usefulness.

• ‘Categorical data is much more difficult to deal with. Processes are time consuming and highly subjective; There are relatively few statistical tools which deal effectively with categorical data.

• The inclusion of valid ‘Categorical’ data, suitably summarised can add both validity and reliability to your study.


HO

PE

‘Descriptive’ Data• Descriptive data is simply a verbal description of

something that happened, or what a respondent did, or a quote from an interview.

• It is the most difficult to deal with effectively. There are methods for extracting summary information from such data (see Grounded Theory: Glazer & Strauss); however, these are not for the novice researcher in IT, as they are very time consuming.

• The best use for such data is for illustration and supporting evidence elsewhere.

• The inclusion of appropriate ‘Descriptive’ data, suitably selected can improve the overall validity of your study.


HO

PE

Types Of DataReferences (SW Library)

• Research methods in education and the social sciences / [Research Methods - Block 5 : Classification and measurement. - Milton Keynes : Open University Press, 1979. - (DE304 : a third .. - 0335074405

• Research methods in education and the social sciences / [Research Methods. - Block 6 : Making sense of data. - Milton Keynes : Open University Press, 1979. - (DE304 : a third level course. - 0335074413


HO

PE

Types of Data References (SW Library)

• Chatfield, Christopher. - Statistics for technology : a course in applied statistics. - 2nd. - London : Penguin Books, 1978. - 0412157500

• Kranzler, Gerald. - Statistics for the terrified / Gerald D. Kranzler, Janet P. Moursund. - Englewood Cliffs, N.J. : Prentice Hall; London : Prentice-Hall International, 1995. – 0131838318

• Pentz, Mike. - Handling experimental data. - Milton Keynes : Open U.P., 1988. - 0335158242

• Salkind, Neil J.. - Statistics for people who (think they) hate statistics. - Thousand Oaks, Calif.; London : Sage, 2000. - 0761916210


HO

PE

6b Extracting Data For Analysis

Data produced by the research process does not have to be numerical in order to be of value. However, statistical techniques work best where the data uses numerical values, or can be converted to numerical scales.


HO

PE

Survey Data

• Questionnaire data should be the easiest data to extract. Ideally the questionnaire will have been designed with the methods of analysis in mind, so that questions are framed in such a way as to facilitate graphical representation and statistical testing.

• Interview Data can be more problematic; you may need to ‘invent’ categories, and to count up instances of different cases, both in the same interview, and with different subjects.


HO

PE

Questionnaire Data Examples

Percentage of Respondents using Broadband: 82% In the survey, 93% of people used their computer to

access the internet, compared to 37% who used it for word-processing, and 7% who used it to develop programs.

87% of Linux users rated their operating system as good or very good, compared to 64% of Windows users.

The average number of hours spent on the computer each week by men was 9.3, compared to 3.7 for women.


HO

PE

Interview Data Examples• 5 out of the 6 interviewees were using Broadband.• During the interview, all interviewees mentioned the positive

benefits of moving to broadband. For example one respondent said “My daughter can use the phone while I am replying to emails”. Altogether there were 25 instances in the interviews where benefits were named, as opposed to only three cases where drawbacks were noted.

• These cases broke down as follows:– 20% of cases referred to accessibility issues– 60% of cases referred to improved speed– 20% of cases referred to improved compatibility

• However, it must be noted that all the drawbacks were cited by a respondent who reportedly used the internet for over 70 hours per week. The other five respondents’ use was an average of 15.7 hours, with totals ranging from 4 hours to 25 hours.


HO

PE

Observational Data

• Pure observational field notes are unlikely to contain numerical data. As with interview data, you may have to ‘invent’ appropriate categories and count up instances or occurrences.

• Observation schedules are easier to deal with. As with questionnaires, these should have been designed with the methods of analysis in mind, so that items are framed in such a way as to facilitate graphical representation and statistical testing.


HO

PE

Observational Data Examples

• Out of the 40 people observed in the study, 36 of them (90%) used the mouse as the first point of contact with the portal.

• After the entry page, 70% visited page 3, 20% visited page 7 and the rest visited pages 2, 4 and 9.

• The mean time taken to access the information by novice users was 163.5 seconds (SD = 24.8 secs); however, those who were subjected to the brief training session were able to access the information in a mean time of 48.3 seconds (SD = 13.9 secs)

• 95% of the users rated the interface good or very good.


HO

PE

Experimental Data

Experiments should have been designed with specific comparisons in mind, where independent variables are identified & controlled, and dependent variables are measured in some way. The methods of analysis should have been identified beforehand, and the whole is susceptible to graphical representation and statistical testing.


HO

PE

Experimental Data Examples

System A

4.5MB: 1.8 secs25.3MB: 7.2 secs185MB: 35.4 secs387MB: 71.8secs875MB: 165.3 secs1.56GB 299.5 secs4.76GB timed out

Using two identical machines, System A using Windows XP and System B using Linux, accessing the website using a 56Mb/sec modem. Download times were:

System B

4.5MB: 1.6 secs25.3MB: 6.4 secs185MB: 36.2 secs387MB: 74.3secs875MB: 155.8 secs1.56GB 287.5 secs4.76GB 889.3 secs


HO

PE

Evaluation Data

Evaluation is unlikely to yield lots of numerical data. Categories on the Evaluation Sheet may well involve numerical values: counts, times, file sizes etc., and these are amenable to graphical representation & testing. Other categories may require judgement on ‘descriptors’, or on quality. It is possible to convert some of these to simple scales (0,1,2,3 etc), and to rank items in order, for example. Yes/No can be encoded as 1 and 0.


HO

PE

Evaluation Data ExamplesThe 5 evaluators rated the 4 packages as follows:

Evaluator 1 2 3 4 5 Best C C D C C

| A D B B D| D B C D B

Worst B A A A A

• Package C performed best overall; it achieved higher grades on 6 out of 7 criteria, and received a total rating of 8.3, compared to ratings of 6.1, 4.5 and 2.8 for packages B, D and A

• The reasons for Package A’s poor performance were cited by the evaluators as: poor readability, confused navigation, and lack of interactivity.

• 4 out of 5 of the evaluators rated the interface design for Package A as poor or very poor.


HO

PE

Data CollectionReferences (SW Library)

• Research methods in education and the social sciences. – Block 4 : Data collection procedures. - Milton Keynes : Open University Press, 1979. - (DE304 ; block 4). - 0335074251

• Research methods in education and the social sciences / [Research Methods. - Block 6 : Making sense of data. - Milton Keynes : Open University Press, 1979. - (DE304 : a third level course. - 0335074413


HO

PE

6c Presenting Data

• This section of the presentation looks at some of the different ways in which you might present data, and the different ways in which you can draw meaning from it.


HO

PE

Presenting Numerical Information

• Frequency Tables

• Cross Tabs

• Time Series Data


HO

PE

Frequency Tables

• In this example the number of errors cited when 3 different websites are run through an html code checker are tabulated.

• It is normal where percentages are given, to quote the actual numbers involved, OR the total number of cases.

Website A

Website B

Website C

Errors Reported 35 12 17

Percentage 54.69% 18.75% 26.56%

The percentages allow an easy comparison to be made on a ‘standard scale’


HO

PE

CrossTabulations• The purpose of a Cross-tabulation is to examine the relationships

between two different variables.• In this example, the number of men and women of different age

groups expressing an interest in Blue Tooth are cross-tabulated.

Under 21 21-30 31-40 41-50 50+Men 87 64 28 31 17

Women 85 32 17 14 4

Under 21 21-30 31-40 41-50 50+

Men (n=227) 38.3% 28.2% 12.3% 13.7% 7.5%

Women (n=152) 55.9% 21.1% 11.2% 9.2% 2.6%

Here the information does not really show what is happening.

Here we have used percentages, and it is now clear that although there is a reduction in interest in both males and females as they get older, it is far more dramatic for females.


HO

PE

Time Series Data

• A Time Series simply logs the value of a particular variable over a period of seconds, days or weeks.

• Because of amount of numerical information, it is very difficult to make sense of time series data unless it is summarised or presented in graphical form.

Time (min) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Unused Sectors 1028 985 972 946 889 825 748 729 719 716 714 693 676 658 643 628 599 579 570 561 545

This data shows the result of an experiment to randomly access websites, and download any .jpg files on the first page onto a hard drive. All we can see here is that the numbers are declining, but we cannot really get a sense of how this is happening


HO

PE

Presenting Graphical Information

• Simple Bar Charts & Histograms

• Comparative Charts

• Scattergrams

• Time Series Data


HO

PE

Simple Bar Charts & Histograms

• Simple graphs can be highly effective in making a point. • This one shows the fact that Package A clearly has been

rated higher than the other two packages.

0

2

4

6

8

Rating

Package A Package B Package C

Average Rating for Packages


HO

PE

Comparative Charts

• All three of these charts display the same information.

• Which makes it clearer?

Under21

21-30 31-40 41-50 50+

MenWomen0

20

40

60

80

100

Frequency

Age group

Number of men and women expressing an interest in Bluetooth

Men

Women


0102030405060708090

100

Under 21 21-30 31-40 41-50 50+

Age group

Frequency

Men

Women


0102030405060708090

100

Under 21 21-30 31-40 41-50 50+

Age group

Frequency

Men

Women


HO

PE

Scattergrams

• These are useful if you are trying to demonstrate a relationship between two variables.

File Size v. Download time (secs)

0.0

50.0

100.0

150.0

200.0

0 0.5 1 1.5 2 2.5

File Size (MB)

Do

wn

loa

d t

ime

s (s

ecs

)

We could add a ‘regression line’ to make the relationship explicit.


HO

PE

Time Series• A variant on the scattergram with lines added, records of what

happened over a period of time to the quantity measured.

Number of Sectors Remaining

0

200

400

600

800

1000

1200

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Minutes into Experiment

Un

use

d S

ect

ors

Here the data shows an almost uniform rate of loss of sectors, minute-by-minute.


HO

PE

Simple Statistics

• Data needs to be summarised using a variety of standard tools: Proportions, expressed in

percentages; Averages, such as means,

medians etc. Measures of variability, such

as ranges, standard deviations etc.

In this example, 3 different websites have been given gradings out of 10 by a panel of 6 experts.

In the summaries, it is clear that A outperforms the others, and C is the worst. However, there is a lot more inconsistency in A’s scores than any of the others.

ExpertWebsite

AWebsite

BWebsite

C

I 8 5 2II 4 4 3III 5 5 3IV 7 6 2V 9 6 2VI 9 7 1

Mean 7.0 5.5 2.2St.Dev 2.10 1.05 0.75


HO

PE

• St. John, P. R.. - Methods of presenting fieldwork data / P.R. St. John, D.A. Richardson. - Sheffield : The Geographical Association, 1989. - 0948512164


HO

PE

6d Statistical Testing

• This section of the presentation explains the principles behind statistical testing, and describes the some of the different types of statistical tests available.

• It also explores probability theory.


HO

PE

Hypothesis Testing






Test Hypotheses; Draw Conclusions

If possible, the final section of your study should include some statistical tests to provide rigour to your arguments.


HO

PE

Probability Theory (1)

• If your data consists of measurements, measured then you can probably assume that you have a ‘Normal’ Distribution, such as IQ scores.

• Lots of people with average IQ scores in the middle, but with very low numbers with High IQs or Low IQs.

http://www.socialresearchmethods.net/kb/sampstat.htm


HO

PE


• Even if your data is not measurement data, you can still make use of ‘Normal’ Distribution techniques, provided that:Your data is counted, and you have high

numbers (over 50 observed in each category)You are working with averages of 10 items or

more.


HO

PE


• Probability tries to measure how likely a particular observed event is, before it occurs.

• For example, if we picked an undergraduate student at random, and asked what is the likelihood of that student winning the lottery, it would be a very low probability event (prob = 0.0000001)

• On the other hand, if we asked what is the likelihood of that student graduating, the probability would be very high. (prob = 0.95)


HO

PE

Random Chance

• In time, all things are possible. Mountains might crumble, pigs might fly, you might win the lottery.

• If we wait around long enough these events will happen, because each event has a small, but finite probability.

• Random chance will dictate the day, the year, or the millennium in which these occur.


HO

PE

Making predictions

• Because of the way that statistics works, we can use the features of data, and the idea of random chance to make predictions.

• If we were to walk into the dining hall, and select a student at random, we could predict the likelihood that:1. The student has an IQ between 85 and 115 (about 70%)2. The student has an IQ over 130 (about 2.5%)

• The first is a high probability event, and that’s what we would normally expect to happen.

• Clearly the second is a low probability event, and therefore we would be very surprised if it occurred.


HO

PE

Hypothesis Testing

• We all work on assumptions every day. Let us suppose that you have come to college on what you think is a normal day, only to find the rooms deserted, no lecturers, no students.

• What is the explanation?

Is it that:

1. You came in on a Sunday by mistake, thinking it was Monday

2. The college has been evacuated due to a leak at a nearby toxic waste incinerator.


HO

PE

Hypothesis Testing

• Most people would immediately suggest (1), because (2) is a very low probability event.

• However, we can use the situation to reason thus:If we assume that Today is Monday, and we

come in and find the college deserted, then the reason must be (2).

However, this is such a low probability event that we would not normally expect to witness it.

This means that something is wrong.It is more likely that our original assumption is

incorrect, and Today is NOT Monday.


HO

PE

Hypothesis Testing

This is formalised into a process which goes like this:• We set up two competing hypotheses (normally

called H0 and H1).• We make observations in data, and calculate the

probability of these observations occurring by random chance. We base this calculation on the assumption that H0 is true.

• If the probability is very low, we reject H0 (the null hypothesis), and accept H1, the alternative hypothesis.


HO

PE

Hypothesis TestingAn Example

Let us set up two competing hypotheses:H0: MSc. Computing is equally attractive to men

and women.H1: MSc Computing is more attractive to one

gender than the other.

We now observe the number of students in a group, and count the number of males and the number of females, and ask what is the likelihood of observing such, if the subject were really equally attractive.

.


HO

PE

Hypothesis TestingAn Example

Suppose in the class there are 8 men2 women

• If H0 is true, then the probability of this event is p=0.01; this is 1%, extremely low .

• We therefore reject H0, and accept H1, concluding that MSc. Computing is more attractive to one gender than the other.


HO

PE

How low can you go?

• The question now comes, how low does a probability have to be in order for you to conclude that the event is not simply occurring as a result of random chance.

• For most circumstances, we would use p<.05 or less than 5%, and we would say such a result is statistically significant

• If p <.005, then the result is highly significant.• If p< .0005, then the result is very highly significant.


HO

PE

What does this all prove?

• Your results may not be statistically significant, that does not mean that they are not true; it simply means that the data you collected was not sufficient to decide things one way or another.

• If your results are significant, then you need to decide whether the interpretation that you are putting in them is actually the correct one; there may be other explanations for the results than the one that you have stated as H1.

• In other words, you should be cautious when you write this up; be analytical, and question everything.


HO

PE

6e Examples of Statistical Tests

• This section looks at some examples of statistical tests that you can undertake in different contexts.

• It distinguishes between Parametric and non-Parametric testing


HO

PE

Parametric Testing (1)

• Parametric Testing assumes that the data you are working with follows a Normal Distribution.

• You can use these techniques if Your data is measured Your data is counted, but there is a large number in the

count Your data is in numerical categories, or categories which

can be translated into numbers, and the overall shape of the data is a ‘bell’ curve.

You are working with averages, based on samples of 10 or more.


HO

PE

Parametric Testing (2)

These are some samples of Parametric Tests• Z-Test

– Single sample: Null hypothesis is that the average of a sample is equal to some theoretical value.

• Paired Sample T-Testing– Two samples: Null hypothesis is that there is no

difference between the two samples• ANOVA

– Three or more samples; null hypothesis is that there is no difference between the samples.

• Correlation– Two measurements taken on the same data; null

hypothesis is that the measurements are not related in any measurable way.


HO

PE

Non-Parametric Testing (1)

• Non-Parametric Testing assumes that the data you are working with does not follow a Normal Distribution.

• You need to use these techniques ifYour data is counted, with low numbers in each

categoryYour data is in categories which cannot be

translated into a numerical scaleData where the overall shape of the data is not a

‘bell’ curve.You are working with medians or other statistical

summary values which are not averages.


HO

PE

Non-Parametric Testing (2)

These are some samples of Non-Parametric Tests:• Chi-Squared Test

– Crosstabs or Contingency Tables showing the effect of two different factors: Null hypothesis that the two factors are independent.

• Mann-Whitney U-test– Two samples: Null hypothesis that there is no difference

between the two samples• Phi Test

– 2 x 2 Crosstab; Null hypothesis is that the factors are unrelated.

• Spearman’s Correlation– Two judges asked to rank a series of items; null hypothesis

is that the rankings are statistically unrelated.


HO

PE

Selected Test Examples

• t-Test

• ANOVA

• Correlation & Regression

• Chi-Squared Test


HO

PE

t-TestThis is used to test whether a the mean of an observed sample is significantly different from an hypothesised value.

• In the example shown, 12 respondents were given a Likert-scaled questionnaire, to determine their attitude towards chat-room ‘behaviour guidelines’.

• A neutral score would be 2.5• A t-test can be used to test the hypothesis:

Ho: Respondents disagree with (or are neutral to) the guidelines

• The t-statistic is calculated to be t = 2.72 at p< .05; so we reject Ho, and conclude that the respondents agree with the guidelines.

Average of Likert Scores

3.42.52.73.13.63.42.82.43.51.92.53.2


HO

PE

ANOVA

This is used to test whether a the means of three or more samples are significantly different from one another.The example above shows the average ratings from 6 different evaluators of three different web-authoring packages, A, B & C on a variety of ten-point scales.We can test Ho: There is no difference between the ratings of the packages, using an ANOVA (Analysis of Variance) test.The results below show a p value of 0.004881We reject Ho, and conclude that the packages ARE different.

The result is highly significant.

Package A

Package B

Package C

8.4 4.7 5.17.5 5.1 4.79.1 6.2 6.26.7 7.5 6.36.2 3.8 5.47.8 6.2 5.6

ANOVASource of Variation SS df MS F P-value F critBetween Groups 16.81333 2 8.406667 7.749667 0.004881 3.682317Within Groups 16.27167 15 1.084778

Total 33.085 17


HO

PE

Correlation & Regression

This is used to determine whether there is a relationship between two variables, and if so, what?

The data collected shows the download times for 10 files of different sizes.

There is a correlation of 0.998864 between the two variables, showing that a very strong relationship exists.

The regression analysis below shows that this can be expressed as:

Download Time = 70.65 x File Size - 1.49

File Size (MB)

Download time (secs)

0.018 1.30.056 3.40.094 5.70.132 7.10.271 15.00.408 28.40.846 60.31.284 87.11.922 128.32.362 170.8

CoefficientsStandard Error t Stat P-value Lower 95%Upper 95%Lower 95.0%Upper 95.0%Intercept -1.49155 1.410594 -1.05739 0.321215 -4.74438 1.76129 -4.74438 1.76129X Variable 1 70.65 1.292702 54.65297 1.39E-11 67.66902 73.63098 67.66902 73.63098


HO

PE

Chi-Squared Test

This is used to determine whether the effects of two or more factors are independent.The data above shows the number of respondents of each gender in different age categories who expressed interest in using Blue Tooth technology within the next 12 months. A Ch-Squared test is used to test the hypothesis: Ho: There is no relationship between age and gender in potential Blue Tooth use.The probability value of the Chi-Squared Statistic is p = 0.00893We would reject Ho, and conclude that there are differences between the age profiles of men and women potential users of Blue Tooth.

Under 21 21-30 31-40 41-50 50+

Men 87 64 28 31 17

Women 85 32 17 14 4


HO

PE

Which test?

• http://research.med.umkc.edu/tlwbiostats/choosetest.html

• http://staff.harrisonburg.k12.va.us/~gcorder/stats_Index_Page.html


HO

PE

Statistical Testing

• http://www.statsoft.com/textbook/stathome.html

• http://faculty.vassar.edu/lowry/webtext.html

• http://www.tufts.edu/~gdallal/LHSP.HTM


HO

PE

Statistical TestingReferences (SW Library)

• Ferguson, George A. - Statistical analysis in psychology and education. - 3rd. - Tokyo : McGraw-Hill Kogakusha, 1971. - m0891134

• Peers, Ian S.. - Statistical analysis for education and psychology researchers. - London : Falmer Press, 1996. - 0750705051

• St. John, P. R.. - Methods of statistical analysis of fieldwork data / P.R. St. John, D.A. Rich. - Sheffield : The Geographical Association, 1989. - 1899085165


HO

PE

6f Analytical Tools

• In order to analyse the data, you will need a package that is capable of arithmetic and other data manipulation, and will allow you to display the data in the form of tables and charts, as well as to perform statistical tests.

• There are several packages that you can use in order to analyse your data, but two of the main ones are:– Microsoft Excel– SPSS


HO

PE

Excel or SPSS?

• Excel will be OK to use if all your data is numeric, and that you have a clear idea of what you want to do, and how to do it. Excel is good at data display, but can only perform limited statistical testing, mainly parametric.

• SPSS is better at statistical testing, is highly versatile and can do lots of clever stuff. You will need this if you intend to analyze questionnaire data, or have lots of complex, categorical data and need to do non-parametric testing.


HO

PE

Using Excel

• Excel is available on all computers, and there is an extensive help facility embedded in it.

• You will probably have worked with it before, so your ‘learning curve’ is fairly gentle.

• If you have only a few items to display, and you are not going to do any heavy statistical testing, then Excel is probably the easiest tool to use.


HO

PE

Example of Excel Use (1)

• You can type data directly into an Excel worksheet.

• Excel has formulae to calculate statistics, such as:= Average(C3:C8)

= StDev(C3:C8)


HO

PE


• This is an example of a chart easily produced from the data on the previous slide using Excel.

III

IIIIV

VVI

0123456789

Ranking

Expert

Expert Rankings

Website A

Website B

Website C


HO

PE


• Excel can produce a range of descriptive statistics very easily.

Website A Website B Website C

Mean 7 Mean 5.5 Mean 2.1667Standard Error 0.8563 Standard Error 0.4282 Standard Error 0.3073Median 7.5 Median 5.5 Median 2Mode 9 Mode 5 Mode 2Standard Deviation 2.0976 Standard Deviation 1.0488 Standard Deviation 0.7528Sample Variance 4.4 Sample Variance 1.1 Sample Variance 0.5667Kurtosis -1.5496 Kurtosis -0.2479 Kurtosis -0.1038Skewness -0.5851 Skewness 0 Skewness -0.3126Range 5 Range 3 Range 2Minimum 4 Minimum 4 Minimum 1Maximum 9 Maximum 7 Maximum 3Sum 42 Sum 33 Sum 13Count 6 Count 6 Count 6


HO

PE

Using SPSS

• SPSS is “Statistical Package for the Social Sciences” and is an industry standard tool for data analysis.

• If you are doing lots of work with numerical data (e.g. analysing questionnaires) then you will need to use SPSS.

• SPSS is available via the F: drive, and it is also available for £5 from the technician in the Psychology Lab (GLB)


HO

PE

SPSS Tutorial


HO

PE

Example of SPSS Use

• Part of a study looks at file upload times under 10 different conditions, using two types of browser: Netscape or Internet Explorer.

• The same circumstances relate to each pair of readings

Netscape Int Exp

13.40 12.9013.80 15.7012.90 11.7015.30 14.6016.40 11.4014.70 14.2015.80 15.3017.10 16.1012.70 11.5011.80 12.10


HO

PE

Setting Up Variables

• First of all we set up descriptors for each of the two variables:– Netscape– Internet Explorer


HO

PE

Entering Values

• Then we Enter the 10 values for each variable, corresponding to the different cases.


HO

PE

Analysing the data

• We can select graphical and numerical summaries


HO

PE

Summaries

Results:

• It looks as if Netscape generally takes longer to upload files than IE: 14.39 secs for NS as opposed to 13.55 secs for IE

• We might try to test this as a hypothesis.


HO

PE

Results of a T-TestWe test:

H0: there is no difference between sample means.

H1: There is a difference between sample means.

P = 0.158.

Unfortunately this is greater than .05, and not low enough to reject the null hypothesis; the result is not significant.


HO

PE

Using SPSS

• http://www.utexas.edu/its/rc/tutorials/stat/spss/spss1/index.html

• http://www.strath.ac.uk/IT/Docs/SPSS/spss.html

• http://www.leeds.ac.uk/iss/documentation/math.html

• http://www.indiana.edu/~statmath/stat/spss/win/giant.html


HO

PE

Using SPSSReferences (SW Library)

• Dancey, Christine P.. - Statistics without maths for psychology : using SPSS for Windows / Christine. - 2nd ed. - Harlow : Prentice Hall, 2002. - 0130336335

• Kinnear, Paul R.. - SPSS for Windows made simple : release 10 / Paul R. Kinnear, Colin D. Gray. - [New ed.]. - Hove : Psychology Press, 2002. - 1841691186

• Pallant, Julie. - SPSS survival manual : a step by-step guide to data analysis using SPSS for Windows. - Buckingham : Open University Press, 2001. - 0335208908

• West, Robert. - Computing for psychologists : statistical analysis using SPSS and MINITAB. - London : Harwood Academic, 1991. - 371865086x

HO

PE

7. Problems with The Research Process


HO

PE

Research Pitfalls

• Finally, we look at some problems that you might encounter which invalidate your research – and how to prevent these.


HO

PE

Some Research Pitfalls (1)Researcher Bias• Problem: Introducing your personal

prejudices or preconceptions into the research, and obtaining invalid results.

• Solution: Counter by an attitude shift: we do not try to prove a hypothesis is true, rather we test whether or not a hypothesis is true, and keep an open mind as to the result.


HO

PE

Some Research Pitfalls (2)The Hawthorne Effect• Problem: Subjects behave differently

because they know they are participating in Research.

• Solution: Use of control group in experiments; concealed observation techniques; surveillance methods; naturalistic data collection methods


HO

PE

Some Research Pitfalls (3)The Halo Effect• Problem: If Subjects rate something as

good or bad overall, they are likely to rate all features of it as good or bad, without discriminating between them.

• Solution: In questionnaires & evaluations, use a variety of similar questions, positively & negatively worded.


HO

PE

Some Research Pitfalls (4)Pseudoscience• Problem: Setting out to prove a

hypothesis by only looking for evidence which confirms it, and disregarding evidence which contradicts it.

• Solution: Explicitly stating what would be disconfirming evidence for a hypothesis, and setting out to look for it.


HO

PE

Some Research Pitfalls (5)Sample Bias• Problem: Getting misleading results,

because the sample on which you have based your analysis is consistently unrepresentative.

• Solution: Identifying the group of people, objects or events you are trying to generalise about, and selecting a sample which has the same characteristics as that group, or is a random selection from it.


HO

PE

Some Research Pitfalls (6)Overgeneralisation• Problem: Making sweeping claims that the

results that you have observed from one or two cases will be present in the population at large.

• Solution: Be cautious in your claims, and use moderate language, such as ‘likely’ or ‘probable’ causes; only claim that your results are ‘significant’ when they are actually statistically significant, after carrying out a statistical test and rejecting the null hypothesis.


HO

PE

Some Research Pitfalls (7)Mistaking Association for Causality• Problem: Thinking that because one variable increases

(or reduces) as another increases (or reduces) that we have found a ‘cause’ of an event. For example, the variable ‘difficulty of navigation’ in a multimedia package may be correlated with the number of buttons. However, it is the underlying navigational structure, which is at the root of the problem, which coincidentally generates the number of buttons.

• Solution: Understanding that correlation is about variation, and that multiple factors often act together to cause particular effects, and seeking such factors.


HO

PE

Some Research Pitfalls (8)Ex Post Facto Hypothesizing• Problem: Deciding on your research

hypotheses after you have seen the data, and noticed some results which might be significant. In any data set, there are bound to be ‘blips’ which are freak results, and you are in danger of basing your results on these.

• Solution: Decide on your research hypotheses and the test you will do before you collect data. That way the tests will be fair, and your results valid.

HO

PE

8. Summary


HO

PE

Summary

Things to consider

• Devising your Research Questions

• Selecting Your Methodology

• Deciding on Your Methods of Analysis

• Some Do’s & Don’ts


HO

PE

Relationship between Elements• To a large

extent, this process is iterative.

• You need a good idea of the data you intend to collect, and how you will use it, before you devise your research questions.








HO

PE

Devising Research Questions

• Your Research Questions should arise naturally out of the Literature Review.

• However, in your Research Project you should only devise questions that you are capable of answering.

• That means when you ‘phrase’ them, you should have a good idea of the methods that you will use to answer them, the data that you will collect, and how that data will be analysed.

• What you should not know,is the answer to those questions!


HO

PE

Selecting your Methodology

• Your methodology should be devised specifically to answer your Research Questions

• It should draw on the methods used by the studies that have formed a part of your Literature Review.

• It should also be created in such a way that the data that it produces is amenable to analysis

• That means you should be able to take advantage of the different forms of analysis, and the different tools available, in order to enhance your research.


HO

PE

Deciding on the Analysis

• The method of Analysis needs to be appropriate to the data collected.

• It also clearly needs to be sufficient in order to answer the Research Questions .

• It should also use methods that have been established by previous researchers that you have highlighted in your Literature Review.


HO

PE

Advice – some Dos

• Plan ahead; think through your methodology and your analysis before framing your Research Questions.

• Include at least one ‘Research Hypothesis’, which you know you are going to test; this adds credibility to your dissertation.

• Spend time looking at how previous researchers have pursued their research; draw on their experience, adapt their methods and modify their research instruments… but don’t forget to credit them in the write-up!

• Consult with your supervisor, take advice.


HO

PE

Advice – some Don’ts

• Do not set out to ‘prove’; set out to ‘investigate’.• Do not try to take on too wide a topic; this is an

MSc. Dissertation, and you should be working in a confined area, but achieving true depth of analysis

• Do not expect to have statistically significant results; most Masters’ dissertations fail to prove anything one way or the other. However, you are expected to know whether the results are statistically significant or not.


HO

PE

… and Finally

• If you are doing a questionnaire, or data analysis, you may need to have specialist advice.

• You can sign up to see me, even though I am not your supervisor.

• However, if you come for a consultation, I expect you to know what you want to do, and have a good idea of how to do it; you should come to me to check that you are on the right lines, not to find out what to do.

• Contact Tony Fleet in FML 213, telephone: 0151-291-3525 or email me at [email protected]


HO

PE

References and Bibliography

Comprehensive accounts of Research methods:• http://www.socialresearchmethods.net/kb/• http://www.socialpsychology.org/methods.htm

Huge list of resources & links:• http://www.geocities.com/Athens/3238/bookmark.htm

Undertaking a Research Project: Guidelines• http://dec.bournemouth.ac.uk/staff/scrowle/

Teaching/FinalYear/ProjectWorkshops/index.html


HO

PE

Research Methods Books in the Sheppard-Worlock:

• Fowler, Floyd J.. - Survey research methods. - 2nd ed. - Newbury Park, Calif.; London : Sage Publications, 1993. - (Applied social research methods series ; v.1). - 0803950489

• Lehman, Richard S. - Statistics and research design in the behavioural sciences. - Pacific

Grove, Calif. : Brooks/Cole, 1991. - 0534138780


HO

PE

Other Research Design Books in the Sheppard-Worlock:

• Handbook of applied social research methods / edited by Leonard Bickman and. - London : SAGE, 1997. - 076190672x

• Human centred methods in information systems : current research and practice. - Hershey; London : Idea Group Pub, 2000. - 1878289640

• Qualitative research : theory, method and practice / edited by David Silverman. - London : Sage, 1997. - 0803976666

• Research design. - A. - Milton Keynes : Open University Press, 1979. - (Social sciences, a third level course research methods in education and Social Sciences ). - 0335074227


HO

PE

Past MSc. Dissertations

• You should examine what other students have done, especially in an area which is akin to yours.

• Remember: if it is on the shelves, it passed!

Examples:• Broadbent, George Ernest. - The role of evaluation in user

interface design : a practical implementation. - Liverpool : University of Liverpool, 1997. - p7270369

• Lomas, Joanne. - The research and design of a computer based training package for children. - Liverpool : University of Liverpool, 2001. - M0002708LO

hope research methods collecting, processing and analyzing data

Documents