Today’s Agenda

• Quick Review• Validity/Reliability/Generalizability • Begin Discussion on Ways to Collect Data– Observations– Field Notes– Documents– Sociogram

Qual or Quant or Both?

• Deals with how/why• Deals with what• Based on empirical evidence• Analyzed with statistical tests• Researcher is the instrument• Small, purposive sample• Valid and reliable

Reliable

• What is meant by reliability?• Demonstration

Validity

• What is meant by valid?• Demonstration

Issue

• Quantitative: use statistical measures to help determine validity and reliability

• Qualitative: we need to convince the reader that the data and analysis are valid and reliable

Reliability

• Are we reporting what is truly being observed• Is what is being observed typical/true behavior• The researcher must come across as unbiased

and not subjective• HOW?– Support observations (videos, tape recordings,

multiple observers)– Address observer bias upfront– How do you fit into the context

Validity

• Matter of fit: does the method of data collection match the problem; are claims supported

• How?– Supporting quotes– Supporting examples– Triangulation

TriangulationA X B

Triangulation

• Used to support validity and reliability• Use multiple data sources (interviews, lesson

plans, observations) to show that the data collected are an accurate/true representation of what you are studying

• Achieved by using multiple researchers (doesn’t work for us), multiple subjects (collecting multiple viewpoints about the same topic)

Example

• Text example on Student Reflections• Let’s take a closer look at the OCEPT project• Interested in whether student teachers were

using reformed based teaching practices in their math and science classes

OCEPT

• Methods of data collection– Three observations with an observation protocol

and global scan notes– Interview– Periodically used multiple observers to find

interrater reliability with the protocol– Multiple student teachers – From multiple programs– (See overhead)

• By presenting data from multiple sources, subjects and researchers, our findings were believed to be an accurate reflection of what was occurring in the classrooms.

Generalizability

• Along with validity and reliability, generalizability is discussed

• Qualitative studies are not designed to be generalizable (what you find is not meant to apply to other settings) so this is a non-issue!

One more note

• If something is valid is it necessarily reliable?• If something is reliable is it necessarily valid?• Consider this simple example:• Assign points for filling in the name on a test.• Is that reliable?• Is that valid?• Can something be reliable and not valid?• Can something be valid and not reliable?

Moving on to data collection

Participant/Observer

Continuum of participation from complete participant to complete observer

Complete Complete Participant Observer

We often fall somewhere in-between

Considerations

• Quality of direct on-site observation (is your presence affecting normal behavior, are you having an observer effect )

• Freedom of access for unbiased observations (can you visit any time, speak to whomever…)

• Intensity of observations (need many/long observations for validity)

• Observer bias• Halo effect

More considerations

• Sampling of data (what is needed from whom/when—example, if examining how a teacher teaches may need to observe planning meetings, department meetings, classroom visits)

• Subtle cues – be alert to unobtrusive signals (is teacher always late to your meetings)

• Will you get too involved?

And still more• Non-participant/observer – minimizes interaction

with subjects being observed• Less likely to become emotionally involved• May be difficult to really get a handle on

perception of subjects• Hard to be unobtrusive

• In cases where you are studying your own students and often your own teachers, you are already involved and so are a participant.

Observations as Data Collection

• Field Notes-description of the setting and what’s going on

• Includes portraits of the subject• Reconstruction of dialog• Description of physical setting• Account of particular events and activities• Description of observer’s behavior• Your reflection of the above

Field Notes

• Include personal reflective notes (O.C. observer comments—Why are these important?)

• Lederman example in text • Overhead of another example

Taping

• May want to audiotape/videotape– Hard to be unobtrusive– Transcription– What it covers/who it covers– Doesn’t get the whole situation (all people, all

gestures, entire class)– Quality is a concern (even volume!)

Practice

• Purpose: Determine good teaching practices to use with ELL (ESOL) students. Observing teachers that have been identified as doing an excellent job with ELL students in their classes

• View the excerpts with an eye toward teaching practices used and student engagement.

Documents• Personal documents of subjects– Lesson plans– Samples of student work– Journals– Power point presentations

• Official– Minutes– reports

• Student records and personal files• Video-audio files• Photographs

Considerations

• Typically work with photocopies

• Note how it was received, collected, when, etc.

• Do you need permission?

Personal Journals

• If you’re doing a case study, have subject keep a journal

• If you’re studying your own students/staff, keep your own journal

• Jot quick notes during/at the end of each class or meeting

• Set aside time later that same day to write a more detailed description of the day’s events

Sociogram

• Useful to understand relationships in a classroom or group

• Ask questions that student responds to using the name of another student

• Example: The purpose is to determine if there is a pattern to student self-selected groups

Sociogram

• Ask students: If you could work with anyone in the class, with whom would you work?

• Note 1st, 2nd, 3rd choices• Graph names and lines showing choices• See OH from Hubbard and Power

Limitations• Can see stars and isolates BUT• Cannot stand alone• Don’t know the reasons behind the students

choices

Closure

• What is triangulation?• Why are field notes important?• What are some key pieces of information

noted in field notes?• What are examples of documents that might

be used as data?