l. o'keefe writing sample - qualitative research narrative

John G. Shedd Aquarium Learning Planning and Evaluation Prepared by: Lindsay O’Keefe, LPE Intern, August 2014 ©Shedd Aquarium

1

RESEARCH NARRATIVE

Qualitative Methods for Evaluating Informal Science Learning

Introduction Evaluators use many different methods in order to evaluate programs, broadly the two main methods are: quantitative methods and qualitative methods. Quantitative methods are defined as, “consist[ing] of research in which the data can be analyzed in terms of numbers” (Best & Kahn, 2006, p. 79). Qualitative methods can, “describe events and persons scientifically without the use of numerical data” (Best & Kahn, 2006, p. 79). Neither one of these methods is better than the other. They both provide important information and insights in different forms and for different purposes. Within the field of evaluation many people have seen the value of using both quantitative and qualitative methods, for these methods help to “improve the validity of their measurements[s]” (Fitzpatrick, Sanders & Worthen, 2004, p. 305). By employing both quantitate and qualitative methods, an evaluator can better understand what is really happening within a program. By using both quantitative and qualitative methods they are able to fill in each other’s gaps and make a more rounded study. This use of mixed method analysis is also known as triangulation (Fitzpatrick, et al 2004). By using triangulation, hopefully everything of interest to the evaluator can be covered and understood. By looking more deeply into the value and use of qualitative methods, more in-depth ways of evaluating a learning program can be achieved. The purpose of this narrative is to inform the Learning Group staff about various qualitative methods that can be used to evaluate the learning programs here at Shedd. The most popular and most common qualitative methods have been included here for reference. What are Qualitative Methods? There are many subcategories within qualitative methods, however, they all have certain ideologies in common that interconnect them. The basic three forms of qualitative data collection are, “(1) in-depth, open-ended interviews; (2) direct observation; and (3) written documents” (Best & Kahn, 2006, p. 247). The many subtypes of qualitative methods help to create specific ways of understanding the general data that can be gathered through non-numerical methods. Some of these specific methods include: interviews, observations, document/content analysis, ethnography, case studies, and focus groups. Qualitative methods are incredibly helpful to getting insights into what a child involved in a learning program is actually thinking they are learning, as it may be different than the intended learning outcome. These thoughts become easily visible through the use of qualitative methods. Why Qualitative Methods? The fields of anthropology and sociology are fields that have long used and long perfected the use of qualitative methods. Sociology also employs the use of some quantitative methods like the use of surveys; however, more than anything the use of qualitative methods are incredibly important insights into a culture or society which is why anthropologists and sociologists use them. Within the field of evaluation qualitative methods can be incredibly helpful for smaller programs that do not have a large enough sample size and as a way to better understand exactly how a student understands the material being presented. Qualitative methods create better insight into exactly how a student interacts with their material. Some advantages to using qualitative methods have to


2

RESEARCH NARRATIVE


do with the fact that they tend to be more flexible than quantitative methods. The evaluator is able to ask open-ended questions and can better probe the participant (Family Health International). For these reasons understanding and utilizing qualitative methods can be incredibly helpful for the field of evaluation and here at Shedd. INTERVIEWS Description Interviews have always been a very important aspect of qualitative methodology. “Qualitative interviews are used for learning the perspectives, attitudes, behaviors, and experiences of others” (Fitzpatrick, et al, 2004, p. 347-8). An evaluator has the ability to ask follow up questions to help participants explain their answer or to get at an idea that may not have been thought to ask in a survey. Interviews allow an evaluator to understand the perspective of the participant. Many more specific forms of qualitative methods include the use of interviews, for example, ethnography and case studies. However, interviews can be a good tool to use by itself without the need to include other methods. There are two forms of interviews, structured and in-depth. When doing a structured interview the interviewer has a strict set of questions that they ask every participant and are instructed not to probe the participants (Frechtling, 1997). This is good when wanting standardization or when there are multiple interviewers and there is a want or need to keep interviews as similar as possible. In-depth interviews are more of a dialogue between the interviewer and participant. Here is where the interviewer wants to probe and ask for more answers to the questions they are asking (Frechtling, 1997). This is good when the evaluator wants a more rounded look into a participant’s thoughts and experiences. Both of these methods have their pros and cons and are good for different purposes. The goal of any interview according to Diamond (1999) “is to elicit a subject’s responses in ways that avoid the imposition of bias on the part of the interviewer” (1999, p. 83). This is incredibly important to remember. When/Where is it Useful Structured interviews are useful as a way to explore a set of thoughts or ideas using “a carefully worded questionnaire” (Fretchling, 2010, p. 60). This type of interviewing is more interested in “obtaining answers to carefully phrased questions (Fretchling, 2010, p. 60). Whereas in-depth interviews explores a visitors thought process and encourages the meaningfulness of their experiences (Fretchling, 2010). The ability to capture an experience through the words of a visitor is one of the great reasons to use in-depth interviews. Interviews, generally, shed light onto experiences that may not be captured otherwise. An evaluator will also receive detailed information and understand more complex matters through the use of interviews (Fretchling, 2010). Whenever there is a need to discuss a complex matter, detailed information or highly sensitive matters in-depth interviews are very useful (Fretchling, 2010). Patton suggests the use of interviews when answering questions like,


3

RESEARCH NARRATIVE


What does the program look and feel like to the participants? To other stakeholders? What do stakeholders know about the project? What thoughts do stakeholders knowledgeable about the program have concerning

program operations, processes, and outcomes? What are participants’ and stakeholders’ expectations? What features of the project are most salient to the participants? What changes do participants perceive in themselves as a result of their involvement in the

project? (Fretchling, 2010 p. 60 (Patton, 1990)). Pros and Cons

Pros Cons Usually yield richest data, details, new insights Expensive and time consuming Permit face-to-face contact with respondents Need well-qualified, highly trained

interviewers Provide opportunity to explore topics in depth Interviewee may distort information through

recall error, selective perceptions, desire to please interviewer

Allow interviewer to experience the affective as well as cognitive aspects of responses

Flexibility can result in inconsistencies across interviews

Allow interviewer to explain or help clarify questions increasing the likelihood of useful responses

Volume of information very large, may be difficult to transcribe and reduce data

Allow interviewer to be flexible in administering interview to particular individuals or in participant circumstance

(Frechtling, 2010, p. 61). Analysis Interview data can be gathered through recordings or through written notes, aka “field notes.” When recording an interview someone needs to then transcribe that data after it has been collected. Transcription is the process of going through the interview and writing down everything that was asked and said. This process requires a lot of time and is important to put in tone of voice when necessary and pauses to questions (Frechtling, 2010). Direct quotes from recorded interviews can often be the most effective way to convey a visitor’s experience (Diamond, 1999). Some training may be necessary to transcribe interviews. When using written notes to document an interview, it can be helpful to have multiple interviewers, one to take notes and the other to ask questions. This way no one is overwhelmed with too many tasks. Note taking is not always as accurate as recordings; however, it takes less time in the long run as there is no need to transcribe the material. This method of note taking is best when there is less time available to collect data (Frechtling, 2010). Regardless of which method is chosen it is incredibly important to use some form of documentation and not simply rely on memory.


4

RESEARCH NARRATIVE


When going over field notes it may also be necessary to code these notes to make them easier to see what categories have been found throughout the interviewing process. It also pulls out all the information that has been gathered. Coding material is one of the unique strengths of qualitative methods. Coding specifically refers to a way of understanding and classifying something so that it is easier to identify commonalities and differences as well as trends. The are two coding methods, emergent coding, which categories items after some preliminary analysis of the data and a priori coding which is establishing categorizes before analyzing the data (Stemler, 2001). For emergent coding,

“two people independently review the material and come up with a set of features that form a checklist. Second, the [evaluators] compare notes and reconcile any differences that show up on their initial checklists. Third the [evaluators] use a consolidated checklist to independently apply coding. Forth, the [evaluators] check the reliability of the coding” (Stemler, 2001).

To do a priori coding evaluators agree on categories and go on to code the data and revisions are made when necessary. Categories are tightened and refined as needed (Stemler, 2001). A priori coding is more strict and planned than emergent coding is. There are programs that help to both transcribe and code field notes, please see An Overview of Content Analysis1 for more information on specific coding software options. Examples in Museums Many museums have taken advantage of the interview, often in conjunction with a survey, or observations to see what the visitors got out of an exhibit and what their thoughts were on said exhibit. Most examples included later in the narrative have some interview component. One specific example is the Summative Evaluation: Mammoth Discovery!2 In-depth interviews were used as a way to understand the visitors experience within the exhibit (Korn, 2012). This is just one of many examples of the use of interviews within evaluation. OBSERVATIONS Description “Observational techniques are methods by which an individual or individuals gather firsthand data on programs, processes, or behaviors being studied” (Frechtling, 1997). Observations can be used both alone and in conjunction with other qualitative methods. In fact, many qualitative methods have some sort of observational component. Observations usually use strict guidelines to record the behaviors of participants (Frechtling, 1997). Observations are valuable because they allow the evaluator to watch their participants and see first-hand what participants are connecting to without the participants translating this information. When/Where is it Useful Using observations during the formative or summative phase is a great way to see whether or not the project is being understood appropriately (Frechtling, 1997). These observations give great

1 http://pareonline.net/getvn.asp?V=7&n=17 2 http://informalscience.org/evaluation/ic-000-000-003-553/Summative_Evaluation_Mammoth_Discovery_

http://pareonline.net/getvn.asp?V=7&n=17

http://informalscience.org/evaluation/ic-000-000-003-553/Summative_Evaluation_Mammoth_Discovery_


5

RESEARCH NARRATIVE


insights into how these projects are being understood by the participants. Shedd has a large number of learning programs and by observing various classrooms there will be a better understanding of the similarities and differences existent within them. Pros and Cons

Pros Cons Provide direct information about behavior of individuals and groups

Expensive and time consuming

Permit evaluator to enter into and understand situation/context

Need well-qualified, highly trained observers; may need to be content experts

Provide good opportunities for identifying unanticipated outcomes

May affect behavior of participants

Exist in natural, unstructured, and flexible setting

Selective perception of observer may distort data

Investigator has little control over situation Behavior or set of behaviors observed may be

atypical (Frechtling, 1997). Analysis There are certain steps that need to be observed in order to ensure that an observation is done well. An evaluator needs to describe the setting in which they are observing. They need to identify the people being observed. They need to document the interactions. Then describe and assess what is going on. These “field notes” are incredibly important to understanding what exactly took place during the observation period (Frechtling, 1997). It can be helpful to have multiple people observing at the same time in order to increase the quality of the data and make sure there is consistency within the observations (Frechtling, 1997). Learning this skill may take some time and may require some further study. Another ways of tracking what is being observed is by creating an ethogram: “An ethogram is a list of the major categories of behavior of which a species is capable” (Diamond, 1999, p. 66). If an evaluator is looking for specific behaviors, for example, students interacting with the animals in the learning labs, this is a great way to make sure those behaviors are being tracked. Creating an ethogram allows the observer to easily document what behaviors are being observed and what they want to observe (Diamond, 1999). Examples in Museums Observations are often used in conjunction with other qualitative methods. The Frye Art Museum used observations in their Summative Evaluation of the Gallery Guide Program3. These observations allowed evaluators to observe how visitors experienced the tours at the museum. These observations were used along with post-tour surveys.

3 http://informalscience.org/evaluation/ic-000-000-007-452/Summative_Evaluation_Of_The_Gallery_Guide_Program_At_The_Frye_Art_Museum

http://informalscience.org/evaluation/ic-000-000-007-452/Summative_Evaluation_Of_The_Gallery_Guide_Program_At_The_Frye_Art_Museum


6

RESEARCH NARRATIVE


DOCUMENT/CONTENT ANALYSIS Description Document/content analysis provides good insights into a setting or group of people that cannot be observed in any other way (Frechtling, 2010). By using document/content analysis an evaluator is able to view thoughts and ideas that would not be revealed through an interview or observation. For evaluation there are many items that can be used for document/content analysis, for example, mission statements, student transcripts, minutes of meetings, official correspondence, presentations, and descriptions of programs among others (Frechtling, 2010). There are also first person documents that can be used, such as diaries, portfolios, photographs, artwork, scrapbooks etc. (Frechtling, 2010). When/Where is it Useful Document/content analysis is useful in understanding interpersonal thoughts of a person that might not be seen otherwise. It also provides crucial insights into the inner thoughts and ideas of a person (Frechtling, 2010). They are also incredibly helpful when trying to make comparisons between participants in a particular program (Frechtling, 2010). Document/content analysis is also useful as the resources needed for the analysis already exist, since the documents have already been created so there is no need to gather more data (Frechtling, 1997). Pros and Cons

Pros Cons Available locally May be incomplete Inexpensive May be inaccurate or of questionable

authenticity Grounded in setting and language in which they occur

Locating suitable documents may pose challenges

Useful for determining value, interest, positions, political climate, public attitudes

Analysis may be time consuming and access may be difficult

Provide information on historical trends or sequences

Provide opportunity for study of trends over time

Unobtrusive (Fretchling, 2010, p. 71). Analysis Many people understand document analysis as doing a word-frequency count (Stemler, 2001). This is not the case, however, it is a part of the process but definitely not all of it. Although it may seem straight forward to do a word-frequency count an evaluator needs to recognize synonyms being used for stylistic reasons or short hand (Stemler, 2001 (Weber, 1990)). Diaries, for example, are not necessarily written for other people to read so the writer may not write in a way intended for analysis, making some information possibly confusing to an evaluator. Coding is also incrediably important in document/context analysis, please see interview analysis section for a more in-depth description of coding.


7

RESEARCH NARRATIVE


Examples in Museums One online learning website, WolfQuest,4 used document/content analysis to help evaluate what their participants were learning and discussing in these online forums. “The goal of the forum analysis was to investigate the considerable amount of social interaction around the site, looking specifically at whether there was evidence of scientific habits of mind being reinforced by the game” (Goldman, Koepfler & Yocco, 2009, p. 9). They specifically looked at topics being discussed. This was a useful way to see whether the players were actually learning science like the producers indented or if they just believed it was a fun and interesting game. ETHNOGRAPHY Description The qualitative method of ethnography is a method that is not easily described as there are many different ways it is utilized. One definition is that ethnography “aims to be holistic by studying ‘naturally occurring human behavior’ through observation” (Roberts, 2009, p. 291). Ethnography also encompasses the terms field work and participant observation as well. Often in literature these three terms are used interchangeably, with ethnography being a more encompassing term tying them all together . It is also described as a “qualitative research approach developed by anthropologists with the purpose of describing an aspect of culture, but is also aimed at learning about the culture factor being studied” (Roberts, 2009, p. 291). With these definitions in mind it is easy to see that ethnography does not fit into a strict box but is open to some interpretation. Ethnography utilizes many different qualitative methods within this larger study, including observations, interviews, and document/content analysis. Within ethnography observing is defined as, “‘individuals, not in isolation, but in relation to their organizations, communities, customs and culture’” (Roberts, 2009, p. 292 (Clark, 2000, p. 44)). The type of observations done within an ethnographic context is similar to just regular observations; however there are certain distinctions that are made. The two types of observations within ethnography are participant and non-participant (Roberts, 2009). “In participant observation the researcher tries to become part of the culture by being in the ‘field,’ rather than from a detached stance as in non-participant observation” (Roberts, 2009, p. 292 (Denscombe, 2003)). “The aim of the non-participant observer, therefore, is to record observations without any interactions within the setting that is being studied” (Roberts, 2009, p 292). Neither one of these methods are better than another they just provide different types of information. When/Where it is Useful Ethnographies are incredibly useful when wanting to understand how people experience the world they are a part of (Roberts, 2009). For example, the environment that is created within the teen learning labs here at Shedd could be examined through the use of ethnography. Ethnographies are used as a way to “produce knowledge,” and this can be about a native tribe as much as a group of teens (Roberts, 2009). It is also a way of being able to make sense of peoples actions through these observations and experiences (Roberts, 2009). It would also be useful when looking for a qualitative method that encompasses many of the other methods.

4 http://informalscience.org/evaluation/ic-000-000-003-244/WolfQuest_Summative_Report

http://informalscience.org/evaluation/ic-000-000-003-244/WolfQuest_Summative_Report


8

RESEARCH NARRATIVE


Pros and Cons Pros for using ethnography include the ability to look at the whole experience of a child in a learning lab (Roberts, 2009). It is able to provide very rich information about the group that is being studied (Roberts, 2009). Cons include the amount of time it takes to do a well-developed ethnography. It is necessary to have staff with some training in this area. There are some limitations to being able to apply one students experience with another’s (Roberts, 2009). It can also be difficult not to become too embedded within what an ethnographer is studying. Ethnographers tend to connect strongly with their subjects (Roberts, 2009).

Participant Observation Non-Participant Observation Pros Researcher is a part of the situation and is

more aware of morale, apathy, and goodwill. May have more credibility.

Can follow a plan of observation.

Cons Might have difficulty recording observations. Might be seen as a threat.

Might be conspicuous, which can affect what is being observed.

(Roberts, 2009). Analysis When doing either participant or non-participant observations, the evaluator will be writing down field notes. To do this please see the observations analysis section for a more in-depth explanation on good observational notes. When it comes to any interviews conducted, they are also analyzed in the same way as a normal interview. Please see the interview analysis section for an in-depth description. Examples in Museums Ethnography was used by the National Science Foundation to evaluate The Cosmic Serpent Story5. For this ethnography they specifically used participant observation. “At least one of the three evaluators– and usually all three evaluators– participated in these modes of interaction with the project team” (Stein, 2011, p. 5). They also made sure to take advantage of field work and used this as another way to observe their subjects (Stein, 2011, p. 5). CASE STUDIES Description The case study is another helpful methodology because it allows the evaluator to do an in-depth analysis of a single unit (Gerring, 2004). John Gerring (2004) wrote that “even among its defenders there is confusion over the virtues and vices of this ambiguous research design” (2004, p. 341). Because case studies are not understood with a universal definition, they can be difficult to define. However, Lynn Davey says that “rather than using large samples and following a rigid protocol to examine a limited numbers of variables, case study methods involve an in-depth,

5 http://informalscience.org/evaluation/ic-000-000-003-452/The_Cosmic_Serpent_Story_Summative_Evaluation_for_Final_Report

http://informalscience.org/evaluation/ic-000-000-003-452/The_Cosmic_Serpent_Story_Summative_Evaluation_for_Final_Report


9

RESEARCH NARRATIVE


longitudinal examination of a single instance or event” (1991). This is the “case” aspect of the “case study.” This is helpful at Shedd to follow the flow and change of a specific learning lab. A full case study will use many different data gathering techniques like document/content analysis, observations, interviews, and surveys among others (Best and Kahn, 2006). A case study tends to present new information and not just back up existing knowledge (Gerring, 2004). There are a couple of different types of case studies that could be used at Shedd. There is the Exploratory Case Study which is used when uncertainty exists about the case being studied (Davey, 1991). It is helpful when trying to explore what the goals, operations, and results of a program are not known yet (Davey, 1991). Program Implementation Case Studies help to weed out if a program's intent matches the implementation (Davey, 1991). “Extensive, longitudinal reports of what has happened over time can set a context for interpreting a finding of implementation variability” (Davey, 1991). This type of case study would be very helpful when it comes to new learning programs at Shedd. Finally there is the Cumulative Case Study which uses and collects data from various different times (Davey, 1991). These case studies build upon past case studies as opposed to starting over with a new one each time (Davey, 1991). It can also use case studies done in the past as well as conducting new case studies. “Retrospective cumulation allows generalization without cost and time of conducting numerous new case studies; prospective cumulation also allows generalization without unmanageably large numbers of cases in process at any one time” (Davey, 1991). This is a good one to have as a long term goal to track the growth and change of a learning program. Each one of these case studies is valuable to Shedd, they simply have different uses and purposes. For a full explanation of each of these methods, and a few others, check out The Application of Case Study Evaluations.6 When/Where is it Useful Case Studies are useful when trying to understand the whole of a unit. “Case studies are particularly useful when the purpose of the evaluation is to describe something – a case – in depth. Very often, evaluations are concerned with exploring the “hows” and “whys” of a program or policy” (Fitzpatrick, 2004, p. 307). For example, answering the question: How does this class work and run? By deciding to do a case study on the Icy Icy Adaptations class you are able to understand specifically how every instructor develops and teaches that specific class. The students and instructors may change, however, the lesson and the learning outcomes should not, so a case study could be a great way to understand how these specific classes run even with a changing variable. Each specific type of case study has a slightly different use as mentioned above.

6 http://pareonline.net/getvn.asp?v=2&n=9

http://pareonline.net/getvn.asp?v=2&n=9

http://pareonline.net/getvn.asp?v=2&n=9


10

RESEARCH NARRATIVE


Pros and Cons Pros Cons Provide a rich picture of what is happening, as seen through the eyes of many individuals

Require a sophisticated and well-trained data collection and reporting team

Allow a through exploration of interactions between treatment and contextual factors

Can be costly in terms of the demands on time and resources

Can help explain changes or facilitating factors that might otherwise not emerge from the data

Individual cases may be over interpreted or overgeneralized

(Fretchling, 2010, p. 73). Analysis Case studies are a combination of multiple qualitative methods and possibly some quantitative methods, these methods include interviews, observation, document/content analysis, and surveys among others. Please see the interview, observation and document/content analysis sections for an in-depth understanding of how to analyze these methods. Examples in Museums A case study method was used to analyze the Animal Eyes7 exhibit in the Museum of Ophthalmology in San Francisco. They wanted to look at this specific exhibit as their unit of analysis. The museum used the case study to help analyze how their visitors interacted with the exhibit to help make improvements for the future. They used observation, open-ended interview questions, and pre/post tests to see visitor knowledge. FOCUS GROUPS Description Focus groups are defined as, “a research technique that collects data through group interaction on a topic determined by the researcher” (Morgan, 1996, p. 130). Focus groups consist of 8 to 12 people who have some connection to what is being evaluated, for example, being a part of a specific learning program at Shedd (Frechtling, 1997). The unique part of a focus group is that a focus group relies on group dynamics allowing the generation of data that would not otherwise occur without group dynamics and discussion (Frechtling, 1997). When/Where is it Useful Focus groups, as other qualitative methods, combine multiple methods of data gathering, including interviews and participant observation (Frechtling , 1997). Focus groups are useful in answering the same types of questions as with in-depth interviews except within a social context (Frechtling, 1997). Some places where focus groups are incredibly helpful are:

Identifying and defining problems in project implementation Identifying project strengths, weaknesses, and recommendations Assisting with interpretation of quantitative findings Obtaining perceptions of project outcomes and impacts

7 http://www.exhibitfiles.org/animal_eyes

http://www.exhibitfiles.org/animal_eyes


11

RESEARCH NARRATIVE


Generating new ideas (Frechtling, 1997). At Shedd focus groups would allow evaluators to understand what students liked and disliked about a certain program and what thoughts and ideas they have to improve them. This is a great way to see the many different viewpoints that each student might have about a program with rebuttals and conversation among the group. The evaluator gets answers to their questions as well as observing how the group interacts with each other. Pros and Cons Pros include the ability to explore thoughts and insights people have along with their behaviors (Morgan, 1996). There is a synergy existent within focus groups that are not existent in other qualitative methods. Focus groups are more efficient. Conducting three focus groups of 10 people each is less time consuming than conducting 30 individual interviews (Morgan, 1996). Cons for the focus group are associated with many of the pros. There are some findings that the moderator can disrupt the flow of a focus group when they ask the participants to explain their train of thought or idea (Morgan, 1996). The moderator has a lot of influence over the productivity of the focus group and must keep the discussion flowing and also make sure no one dominates the discussion (Frechtling, 1997). For these reasons the moderator needs to have some training in leading a focus group or group discussion in order to make them effective. Also only certain topics are acceptable for group discussion since group interaction is essential for a focus group (Morgan, 1996). This may eliminate sensitive/personal topics from a focus group discussion. Analysis Similarly for interviews an evaluator can either record a focus group session or they can take down notes. Again it is helpful to have two people if you are planning on taking notes so as not to split attention from the moderator to be required to both ask questions and record the data (Fretchling, 1997). A focus group is not the same as an interview; however they do use the same method for analyzing the data collected. Please see the interview analysis section for a more in-depth understanding of this type of analysis. Examples in Museums The Cosmic Serpent Story8 not only utilized ethnography, as mentioned previously, but also utilized focus groups as well. This was done to help the evaluators gather feedback from their participants to see how they felt about their experience (Stein, 2011). They wanted to understand exactly what their participants got out of the experience, so the use of a focus group was very helpful for these evaluation efforts.

8 http://informalscience.org/evaluation/ic-000-000-003-452/The_Cosmic_Serpent_Story_Summative_Evaluation_for_Final_Report

http://informalscience.org/evaluation/ic-000-000-003-452/The_Cosmic_Serpent_Story_Summative_Evaluation_for_Final_Report


12

RESEARCH NARRATIVE


References Best, J.W., & Kahn, J.V. (2006). Research in Education 10th Edition. Boston: Pearson Education Inc. Bogdan, R.C. & Bilken, S.K. (2003). Qualitative Research for Education An Introduction To Theorie and Models Davey, L., (1991). The application of case study evaluations. Practical Assessment, Research & Evaluation,

2(9). Diamond, J., (1999). The Pratical Evaluation Guide: Tools for Museums & Other Informal Educational Settings.

Lanham, MD: AltaMira Press. Fitzpatrick, J.L., Sanders, J.R. & Worthen, B.R. (2004). Program Evaluation: Alternative Approaches and

Practical Guidelines 3rd Edition. Boston: Pearson Education Inc. Frechtling, J., Sharp, L. & Wesat, (Ed.). (1997). User-Friendly Handbook for Mixed Method Evaluations.

National Science Foundation. Frechtling, J., Westat (2010). The 2010 User-Friendly Handbook for Project Evaluation. National Science

Foundation. Gerring, J., (2004). What Is a Case Study and What Is It Good for?. American Political Science Review, 98(2)

341-354. Goldman, K.H., Koepfler, J. & Yocco, V. (2009). WolfQuest Summative Evaluation Full Summative Report.

Minnesota Zoo. Korn, Randi and Associates. (2012). Summative Evaluation: Mammoth Discovery!. San Jose, CA: Children’s

Discovery Museum. Morgan, D.L., (1996). Focus Groups. Annual Review of Sociology, 22 129-152. Randall, et al., (2007). Ethnography and How to Do it. Fieldwork for Design (pp. 169–197). Springer. Roberts, T., (2009). Understanding ethnography. British Journal of Midwifery, 17(5) 291-293. Rosen, M., Kinsely, R.P., & Nelson, P. (2012), Formative/Summative Evaluation of the Gallery Guide Program

At The Frye Art Museum. New Directions Partner Institutions. Stein, J., Valdez, S., & Jones, E. (2011). The Cosmic Serpent Story: Summative Evaluation for Final Report. Stemler, S., (2001). An Overview of Content Analysis. Pratical Assessment, Research & Evaluation. 7(71).

l. o'keefe writing sample - qualitative research narrative

Documents