TSL3133 Data collection methodsChecklists & Interviews

By Norherani Moning27/1/2015

Quantitative methods – Qualitative methods

Quantitative Qualitative

SurveysQuestionnaires

Focus groups

Tests Unstructured interviews

Existing databases Unstructured observations

Often, it is better to use more than one method….

• Mixed methods for action research−Log of activities and participation−Self-administered questionnaires

completed after each workshop−In-depth interviews with key

informants−Observation of workshops−Survey of participants

Instrument

The generic term that researchers use for a measurement device (survey, test, questionnaire, etc.). To help distinguish between instrument and instrumentation, consider that the instrument is the device and instrumentation is the course of action (the process of developing, testing, and using the device).

InstrumentsResearcher completed Instruments

Subject-completed instruments

Interview schedule/guides QuestionnairesRating scale Self checklistTally sheets Attitude scalesFlowcharts Personality inventoriesPerformance checklist Achievement/aptitude testTime and motion logs Projective devicesObservation forms Sociometric devices

Observation…

• Is watching people, programs, events, communities, etc.

• Involves all 5 senses: sight, hearing, smell, touch, and taste

− observation includes more than just “seeing”

Observation is used…

• To provide information about real-life situations and circumstances

• To assess what is happening• Because you cannot rely on participants’

willingness and ability to furnish information

When is observation useful?• When you want direct information• When you are trying to understand an

ongoing behavior, process, unfolding situation, or event

• When there is physical evidence, products, or outcomes that can be readily seen

• When written or other data collection methods seem inappropriate

Observations•Advantages

– Most direct measure of behavior

– Provides direct information

– Easy to complete, saves time

– Can be used in natural or experimental settings

•Disadvantages– May require training– Observer’s presence

may create artificial situation

– Potential for bias– Potential to overlook

meaningful aspects – Potential for

misinterpretation– Difficult to analyze

Observation – Purpose, benefits • Unobtrusive • Can see things in their natural context• Can see things that may escape conscious

awareness, things that are not seen by others• Can discover things no else has ever really paid

attention to, things that are taken for granted • Can learn about things people may be unwilling to talk

about• Inconspicuous – least potential for generating

observer effects• Least intrusive of all methods• Can be totally creative – has flexibility to yield insight

into new realities or new ways of looking at old realities

Observation – Limitations1. Potential for bias

• Effect of culture on what you observe and interpret

2. Reliability• Ease of categorization

•Usually you do not rely on observation alone; combine your observations with another method to provide a more thorough account of your program.

Observation – Ethical issues• Unobtrusiveness is its greatest strength;

also potential for abuse in invasion of privacy

• Can venture into places and gather data almost anywhere

• Covert – overt− Always consider ethics and human subjects

protection.

Types of observation

• Structured Unstructured

Observing what does not happen may be as important as observing what does happen.

Looking for

Looking at

Steps in planning for observation• Determine who/what will be observed.• Determine aspects that will be observed

(characteristics, attributes, behaviors, etc.).• Determine where and when observations

will be made.• Develop the observation record sheet.• Pilot test the observation record sheet.• Collect the information.• Analyze and interpret the collected

information.• Write up and use your findings.

Who/what to observe • People (individuals, groups, communities)

− Characteristics − Interactions− Behaviors− Reactions

• Physical settings• Environmental features• Products/physical artifacts

Observation – Example If you want information about… You would record…

Who uses a particular service

Total number of users broken down by gender, age, ethnicity, etc.

Interactions between youth and adults

# and types of questions asked by each

Neighborhood safety ???

What to observe − Example• Exhibit on tobacco use at a county fair

• Information needed – • Number of youth who visit the exhibit:

age, gender, cultural background

Can the information be observed accurately? e.g., gender may be more obvious than age or cultural background.

Will the observer affect the situation?

Recording your observations •Observations need to be recorded to be credible. You might use:

• Observation guide• Recording sheet• Checklist• Field note• Picture• Combination of the above

Observational rating scales• Written descriptions –

written explanations of each gradation to observe

• Photographs – series of photos that demonstrate each of the grades on the rating scale

• Drawings, sketches, etc. – other visual representations of conditions to be observed

Practice• For today’s class what observational

data could we collect that would tell us …• whether learning is occurring• the characteristics of attendees • whether the setting is conducive to

learning• whether the materials are easy to use

Instruments

Checklists

• A list of skills or behaviours to be checked off as they are observed.

• Checklists are often presented as lists with small checkboxes down the left hand side of the page. A small tick or checkmark is drawn in the box after the item has been completed.

Checklists - advantages

• to reduce failure by compensating for potential limits of human memory and attention

• to ensure consistency and completeness in carrying out a task.

Checklists…

research• Observation• Interview – many data collectors

Open-ended ChecklistsEg: practitioner wants to research on her own

teaching performance:• Colleague to collect data of the practitioner • Aim – to capture the full array and distribution of

teacher’s instructional activities and students’ classroom experiences on a typical school day

• Eg.-record a “snapshot” of data every 30 seconds during 70 separate five-minute classroom visits.

Open-ended checklist of observed teacher activities

Activity No of snapshots where observed

Percentage of total snapshots365

Responding to or helping individual students

112 18%

Asking questions 100 16%

Lecturing 91 14%

Paperwork (at desk) 42 7%Active monitoring (looking for students needing help)

41 6%

Getting materials 38 6%

Administering tests 27 4%

Management (attendance, organisational details)

27 4%

Audiovisual equipment set-up 25 4%

Open-ended Checklists…• new category of student or teacher behavior was

observed - added it to the list• previously observed behaviour was observed a

second or third time - simply noted it with another slash mark.

• Data collector need not be concerned that the list of categories is becoming too long.

• Categories can always be combined

Predefined Checklists• Pre-determined list of criteria to be checked or

observed• Planned in advance• Inter-rater reliability– multiple collectors • Definitions must be unambiguous and observable –

many data collectors• Members – meeting to define terms and achieve

consensus on eg. observing students’ performance in class using Bloom’s taxonomy

Predefined checklist :cognitive level of lesson in a high school

Activity Number of Times Observed

Comprehending new information 21Applying knowledge 19Analysing information 6Synthesising information 12Evaluating information 6

Performance Checklists• Consist of a list of behaviours that make up a

certain type of performance• Used to determine whether or not an

individual behaves in a certain (usually desired) way when ask to complete a particular task

Performance checklist sample

1. Takes slide ____2. Wipes slide with lens paper ____3. Wipes slide with cloth ____4. Wipes slide with finger ____5. Moves bottle of culture along the table ____ 6. Places drop or two of culture on slide ____

Self-checklists

• A list of several characteristics or activities presented to the subjects of a study

• To diagnose or to appraise own performance

Self-checklist sample• Date_________ Name___________• Instructions: Place a check (√) in the space provided for

those days, during the past week, when you have participated in the activity listed.

• Mon Tue Wed Thur Fri

1. I participated in class discussion.2. I did not interrupt others while they were speaking.3. I encouraged others to offer their opinions.

•

Language FeaturesLanguage FeaturesThe writer:

Always Sometimes Not Yet Teacher’s Comments

uses a formal and objective style

demonstrates consistent use of tense

demonstrates consistent use of singular and plural generic participants

Research Interviews

• 2-person conversation initiated by the interviewer for the specific purpose of obtaining research-relevant information and focused by him on content specified by research objectives of systematic description, prediction, or explanation.

(Cannell & Kahn in Cohen, Manian & Morrison 2011)

Interviews

• Enables researchers/interviewers to gather data through direct verbal interaction

• Answers to questions are recorded by researcher

Interviews vs everyday conversation

• specific purpose• often question-based (questions asked by the

interviewer)• Interviewer may express ignorance• Interviewee provide explicit responses usually

in detail• Constructed and usually a specifically plan

event rather than naturally occurring situation

Types of interviews

• Interviews differ in the openness of their purpose, their degree of structure, the extent to which they are exploratory or hypothesis-testing, whether they seek description or interpretation, or whether they are largely cognitive-focused or emotion-focused.

Structured interviews

• Respondent is asked a series of pre-established questions with pre-set response categories

• All respondents receive the same questions in the same order, delivered in a standardised manner

• There is little room for variation in response• Interviewer – plays neutral role

Structured interview e.g.

Writing strategies• Are you a good writer? Why?• What makes you a good writer? Why?• When you are writing and you get stuck, what do

you do?• When you don’t know how to spell a word, what do

you do?• What do you like best about your own writing?

Semi-structured interviews

• use an interview guide with some questions developed in advance

• Use open-ended questions• allow the interviewer to stray from the

interview guide by asking follow-ups as the interviewer believes appropriate

Unstructured interviews• Open situation, greater flexibility and freedom • Although the research purposes govern the questions

asked, their content, sequence, and wording are entirely in the hands of the interviewer/interviewee

• need only a checklist of topics to be covered • There is no order and no script• The interaction between the participant and the

researcher is more like a conversation than an interview.

Unstructured / Non-directive interviews (cont…)

• Minimal direction or control exhibited by interviewer

• The freedom the respondent has to express her subjective feelings as fully and as spontaneously as she chooses or is able

Advantages• Flexible• Can use non verbal cues for more information• Interviewer can clarify any questions that are

obscure – gets behind immediate level of response

• Can be used with little fore-knowledge of the situation

• Situational - gets respondent’s view framework/ interpretation

Disadvantages• Not standardised• Not replicable• Time consuming, expensive• Difficult to get co-operation from respondents• Presence of interviewer (researcher) may

inhibit respondents from saying what they really think

• Results difficult and time consuming to analyse

Probing• Interview question:

“What did you like best about this program?”• Response: “I liked everything.” • Probe 1: “What one thing stood out?”• R: “Being with my friends.”• Probe 2: “What about the program activities?”• R: “I liked it when we worked as a team.”• Probe 3: “How come?”• R: “It was neat to hear each other’s perspectives. I

heard some things I hadn’t considered before.” • Probe 4: “What is one thing that you learned?”

Interviewing tips• Keep language pitched to that of

respondent• Avoid long questions • Create comfort• Establish time frame for interview• Avoid leading questions• Sequence topics• Be respectful• Listen carefully

Recording responses• Write down response• Tape record• Key in on computer• Work in pairs• Complete notes after interview

Tutorial Task• Refer to the previous table that you have

created for Dr. Zaira’s tutorial. You have identified your focus, topic, problem statement, research questions etc.

• Now based on all these details plan your method and create your instruments. Make sure that your instruments will provide data that answers your Research Questions.