Telling the Difference Between Telling the Difference Between Baloney and Serious Claims Baloney and Serious Claims

About What WorksAbout What Works

Martin KozloffMartin Kozloff

State departments of education, State departments of education, school districts, schools, and school districts, schools, and teachers select and use teachers select and use curriculum materials, teaching curriculum materials, teaching methods, and assessment methods, and assessment instruments. These cost a lot of instruments. These cost a lot of money. Using them takes time money. Using them takes time and effort---that can’t be and effort---that can’t be replaced.replaced.

But do these things work?But do these things work?

Do assessment instruments provide Do assessment instruments provide information that tells teachers exactly what information that tells teachers exactly what their students know (e.g., algebra pre-their students know (e.g., algebra pre-skills) so that teachers can plan skills) so that teachers can plan instruction? instruction?

Do teaching methods clearly communicate Do teaching methods clearly communicate knowledge so that students master the knowledge so that students master the skills taught? skills taught?

Do curriculum materials (e.g., in beginning Do curriculum materials (e.g., in beginning reading, math, science, and history) provide reading, math, science, and history) provide a comprehensive and logical sequence of a comprehensive and logical sequence of knowledge units; do the materials provide knowledge units; do the materials provide activities to build fluency, generalization, activities to build fluency, generalization, and retention; do the materials have built-and retention; do the materials have built-in assessments so that teachers can track in assessments so that teachers can track their students’ progress? their students’ progress?

DO THEY WORK? DO THEY WORK?

You wouldn’t give your children You wouldn’t give your children medications and THEN SEE if the medications and THEN SEE if the medications work. You wouldn’t medications work. You wouldn’t eat a bunch of broccoli and eat a bunch of broccoli and THEN see if it’s full of bugs. You THEN see if it’s full of bugs. You wouldn’t buy a car and THEN see wouldn’t buy a car and THEN see if the wheels fall off. if the wheels fall off.

YOU WANT TO KNOW IF THE YOU WANT TO KNOW IF THE MEDICATIONS, FOOD, AND CAR MEDICATIONS, FOOD, AND CAR ARE SAFE AND EFFECTIVE ARE SAFE AND EFFECTIVE BEFOREBEFORE YOU USE THEM. YOU USE THEM.

But how can you know this?But how can you know this?The answer is: The answer is: 1. You read articles in journals…1. You read articles in journals…2. You go to conferences and workshops…2. You go to conferences and workshops…3.3. You check out websites on the You check out websites on the

internet…internet…4. You hear your colleagues….4. You hear your colleagues….

And all of these make CLAIMS about the And all of these make CLAIMS about the effectiveness of curriculum materials, effectiveness of curriculum materials, teaching methods, and assessment teaching methods, and assessment instruments. But are these claims instruments. But are these claims BELIEVABLE? Would you ACT on them? BELIEVABLE? Would you ACT on them? What do you have to read or hear about What do you have to read or hear about materials, methods, and assessments materials, methods, and assessments before you feel CONFIDENT about USING before you feel CONFIDENT about USING them? Can you tell the difference between them? Can you tell the difference between claims that sound like baloney vs. claims claims that sound like baloney vs. claims that sound like the real thing---honest, that sound like the real thing---honest, accurateaccuratecredible? credible?

Here are things to consider.Here are things to consider.1.1.Are the claims too good to be true; Are the claims too good to be true; exaggeratedexaggerated; ;

puffed up? Or do they soundpuffed up? Or do they sound reasonable reasonable? ? 2.2.Is the language flowery and vague? Does it Is the language flowery and vague? Does it

appeal to your feelings and not to your appeal to your feelings and not to your intelligence? intelligence?

3. Does the research article, persuasive essay, or 3. Does the research article, persuasive essay, or description (e.g., of materials) communicate that description (e.g., of materials) communicate that the author is intellectually honest. That is, does the author is intellectually honest. That is, does the author communicate that he or she is NOT the author communicate that he or she is NOT certain that he or she is right; that the author in certain that he or she is right; that the author in fact DOUBTS that he or she is right; and that the fact DOUBTS that he or she is right; and that the author has LOOKED for evidence (data) that author has LOOKED for evidence (data) that SHOWS that he or she is WRONG? skepticism, SHOWS that he or she is WRONG? skepticism, doubt, and intellectual honesty----which imply doubt, and intellectual honesty----which imply that an author must try to DISconfirm what he or that an author must try to DISconfirm what he or she believes or wants to be true? [“Nonresearch” she believes or wants to be true? [“Nonresearch” is a set of claims that do NOT rest on skepticism, is a set of claims that do NOT rest on skepticism, but presume truth.]but presume truth.]

4.4.Background coverage of the topic: literature Background coverage of the topic: literature review.review.

Let’s take these one at a time.Let’s take these one at a time.

Are the claims too good to be Are the claims too good to be true; true; exaggeratedexaggerated; puffed up? ; puffed up?

Or do they soundOr do they sound reasonable reasonable? ?

An ad on TV says,An ad on TV says,

““USE Cocoa Creme. You’ll have a USE Cocoa Creme. You’ll have a healthy complexion in 10 minutes. healthy complexion in 10 minutes.

All natural ingredients.”All natural ingredients.”

You say, “Oh, sure! Healthy You say, “Oh, sure! Healthy complexion. I bet your face falls off in complexion. I bet your face falls off in 10 minutes.”10 minutes.”In other words, this claim smells like In other words, this claim smells like baloney. Why? Because your Baloney baloney. Why? Because your Baloney Detector is telling you:Detector is telling you:

1.1. “That claim is exaggerated? Ten “That claim is exaggerated? Ten minutes? No way!!”minutes? No way!!”

2.2. “What does ‘healthy complexion’ “What does ‘healthy complexion’ mean? Will my skin BE healthy or just mean? Will my skin BE healthy or just LOOK healthy because Cocoa Crème is LOOK healthy because Cocoa Crème is like paint that covers all the scabs?”like paint that covers all the scabs?”In other words, you aren’t fooled by ads In other words, you aren’t fooled by ads that promise what is very unlikely, and that promise what is very unlikely, and that give no BELIEVABLE EVIDENCE to that give no BELIEVABLE EVIDENCE to support the inflated promises.support the inflated promises.

Is the language flowery and vague? Is the language flowery and vague? Does it appeal to your feelings and not Does it appeal to your feelings and not to your intelligence?to your intelligence?Is the following claim believable?Is the following claim believable?Cough-No-More is a holistic throat-Cough-No-More is a holistic throat-friendly product made from all natural friendly product made from all natural ingredients by customer-centered ingredients by customer-centered health facilitators. health facilitators.

Cough-No-More will empower you to Cough-No-More will empower you to create a balanced, oxygen-rich create a balanced, oxygen-rich environment, and to share with others environment, and to share with others in your breathing community in your breathing community authentic inhaling experiences.authentic inhaling experiences.

You say, “Oh, sure! I’ll run right out and You say, “Oh, sure! I’ll run right out and buy a case. [sarcasm] But does it WORK?”buy a case. [sarcasm] But does it WORK?”

This vague, flowery language appeals to This vague, flowery language appeals to your feelings, not to your intelligence. your feelings, not to your intelligence. The advertiser is using feel-good words The advertiser is using feel-good words (holistic, authentic, natural) that distract (holistic, authentic, natural) that distract you from asking questions that you MUST you from asking questions that you MUST ask if you are concerned with health!ask if you are concerned with health!

Does Cough-No-More work?Does Cough-No-More work? What exactly is it made of?What exactly is it made of? How is it made?How is it made?

The ad doesn’t tell you. The ad doesn’t tell you.

It’s too busy trying to sell you.It’s too busy trying to sell you.

As you know, the field of education has a lot of As you know, the field of education has a lot of words that make persons words that make persons feel goodfeel good.. ““We use authentic literature.” [What exactly We use authentic literature.” [What exactly does that mean?]does that mean?]““We use child-centered practices.” [What We use child-centered practices.” [What else?]else?]““We have a seamless curriculum.” [Does that We have a seamless curriculum.” [Does that mean anything?]mean anything?]““We use holistic writing assessment.” [That We use holistic writing assessment.” [That sounds good, but it usually means that you sounds good, but it usually means that you don’t look at the details of writing.]don’t look at the details of writing.]““We use only best practices.” [How can anyone We use only best practices.” [How can anyone besides G-d know what’s best? Besides, best besides G-d know what’s best? Besides, best for whom? Best for ALL subject matter?]for whom? Best for ALL subject matter?]““Our instruction is naturalistic.” [In contrast Our instruction is naturalistic.” [In contrast to what? to what? SuperSupernaturalistic?]naturalistic?]

Some persons (NOT YOU!) hear and read these Some persons (NOT YOU!) hear and read these words and PRESUME that the words mean words and PRESUME that the words mean something; they presume that the persons who something; they presume that the persons who write and say the words are talking about REAL write and say the words are talking about REAL things; and they presume that these words things; and they presume that these words should be FOLLOWED.should be FOLLOWED.““A seamless curriculum!!A seamless curriculum!! That sounds That sounds important. I’d better use those curriculum important. I’d better use those curriculum materials, too. I don’t want my students to have materials, too. I don’t want my students to have a curriculum with seams in it. I don’t know what a curriculum with seams in it. I don’t know what seams are, but it doesn’t sound good.”seams are, but it doesn’t sound good.”

““Child-centered practices!Child-centered practices! Yes, of course!!! I’d Yes, of course!!! I’d better use those practices. Children are better use those practices. Children are important.”important.”““Authentic literature. I have no idea what that Authentic literature. I have no idea what that means, but I’d better use those books, too. I means, but I’d better use those books, too. I wouldn’t want my students to read wouldn’t want my students to read ININauthentic authentic literature.”literature.”

Are these words really any different from Are these words really any different from ads that talk about using all-ads that talk about using all-natural natural ingredientsingredients; products made from ; products made from pure pure waterwater (in contrast to what? sewage?); a (in contrast to what? sewage?); a carpet cleaning carpet cleaning systemsystem (Puhleeze! it’s a (Puhleeze! it’s a vacuum clearer with lots of hoses); soap vacuum clearer with lots of hoses); soap that is that is certified free of harmful chemicalscertified free of harmful chemicals (unless you eat it)? No, they serve the (unless you eat it)? No, they serve the same purpose----they are selling you same purpose----they are selling you something.something.

In education, when authors use vague and In education, when authors use vague and flowery words that appeal to feelings and flowery words that appeal to feelings and not to your intelligence, they often are not not to your intelligence, they often are not ABLE to be more precise and objective. ABLE to be more precise and objective. They do NOT have BELIEVABLE They do NOT have BELIEVABLE EVIDENCE to support their claims about EVIDENCE to support their claims about their methods or materials.their methods or materials.

Is it a good idea to use Is it a good idea to use methods and materials that methods and materials that are presented as if are presented as if selectingselecting and and using using them were no more them were no more important than buying cough important than buying cough drops?drops?

Intellectual honesty. Testing the null Intellectual honesty. Testing the null hypothesis.hypothesis. Intellectually honest researchers, Intellectually honest researchers, writers, and advertisers (e.g., writers, and advertisers (e.g., companies that produce materials and companies that produce materials and training programs and teaching training programs and teaching methods) DO NOT pass off their methods) DO NOT pass off their opinions without supporting evidence. opinions without supporting evidence. Moreover, they do NOT simply look Moreover, they do NOT simply look for evidence that supports what they for evidence that supports what they believe. You can always find students believe. You can always find students who learn something even when the who learn something even when the materials are horrible. Those few materials are horrible. Those few students do not show that the students do not show that the materials are effective.materials are effective.

Intellectual honesty. Testing the null Intellectual honesty. Testing the null hypothesis.hypothesis.

The intellectually honest researcher, writer, The intellectually honest researcher, writer, and company STARTS with the idea that he and company STARTS with the idea that he or she is WRONG. And them conducts or she is WRONG. And them conducts research (collects evidence) that can show research (collects evidence) that can show either (1) that the researcher’s belief IS either (1) that the researcher’s belief IS false (e.g., the researcher believes that a false (e.g., the researcher believes that a program will teach children, but the data program will teach children, but the data show that the program does not. Therefore, show that the program does not. Therefore, the belief is false.); or (2) show that the the belief is false.); or (2) show that the researcher’s belief is NOT false (e.g., researcher’s belief is NOT false (e.g., children do learn; therefore, the children do learn; therefore, the researcher’s belief is not false.).researcher’s belief is not false.).

Null HypothesisNull Hypothesis Is the research or report based on the null Is the research or report based on the null hypothesis—an intellectually honest hypothesis—an intellectually honest position? Here is an example.position? Here is an example.

““I believe program X works, but I’m going I believe program X works, but I’m going to assume that it to assume that it doesn’tdoesn’t work and I’m work and I’m going to collect data to try to going to collect data to try to show that it show that it doesn’t workdoesn’t work. If the data do NOT show . If the data do NOT show that X does NOT work, I will conclude that that X does NOT work, I will conclude that maybe it does workmaybe it does work. Maybe.” . Maybe.”

Or, on the contrary, is the author is trying Or, on the contrary, is the author is trying to persuade readers to join him in to persuade readers to join him in accepting a method, or does the author accepting a method, or does the author already believe he knows the truth or already believe he knows the truth or knows what works, or is the author not knows what works, or is the author not very interested in the possibility that he is very interested in the possibility that he is wrong?wrong?

This is what you DON’T want to see.This is what you DON’T want to see.

““Good readers do NOT sound out words. Good readers do NOT sound out words. Good readers use context to Good readers use context to predictpredict what what words say. I will words say. I will support support this statement this statement with examples.” with examples.”

The point is The point is notnot to support what you to support what you believe, but to TEST what you believe, but to TEST what you believe—to believe—to try to FALSIFY what you believe.try to FALSIFY what you believe. The The intellectually honest researcher looks for intellectually honest researcher looks for examples that do NOT support the belief examples that do NOT support the belief or hypothesis. The intellectually honest or hypothesis. The intellectually honest researcher looks for good readers who do researcher looks for good readers who do NOT guess. Why? Because you can NOT guess. Why? Because you can always find evidence to support the always find evidence to support the stupidest ideas. Hitler could “prove” he stupidest ideas. Hitler could “prove” he was a kind person by reporting how often was a kind person by reporting how often he petted his dog.he petted his dog.

Here’s another example of what you Here’s another example of what you don’t want to see.don’t want to see.

An author only reports studies that show An author only reports studies that show one thing (that Program X is associated one thing (that Program X is associated with high achievement). with high achievement).

The author concludes that Program X The author concludes that Program X produces high achievement. However, produces high achievement. However, there is some research showing that there is some research showing that Program X did not produce high Program X did not produce high achievement. But the achievement. But the author did not author did not report THAT research. This is called report THAT research. This is called “cherry picking.” “cherry picking.” It’s also called It’s also called not telling the whole truth.not telling the whole truth.

In contrast, here is what you In contrast, here is what you DO want to see. DO want to see.

The author distrusts findings The author distrusts findings that support what she believes that support what she believes (Program X works), and she (Program X works), and she continues to try to show that continues to try to show that she is RIGHT to distrust her she is RIGHT to distrust her beliefs. She replicates the beliefs. She replicates the research and continues to try research and continues to try to show that Program X does to show that Program X does NOT work.NOT work.

For example,For example,An author conducts research on Math An author conducts research on Math Program X in one school (suburban, middle Program X in one school (suburban, middle class). Kids who got Program X learned a class). Kids who got Program X learned a lot of math. The author says, “It probably lot of math. The author says, “It probably won’t work again.” won’t work again.”

She tests the program again in a very She tests the program again in a very similar school. Same results. “Okay, but it similar school. Same results. “Okay, but it won’t work in a poor, urban school.” Same won’t work in a poor, urban school.” Same results. “Okay, that was a fluke. It won’t results. “Okay, that was a fluke. It won’t happen again.” happen again.”

She tests it in another poor, urban She tests it in another poor, urban school. Same results. “Okay, but it won’t school. Same results. “Okay, but it won’t work with Hispanic kids.” It does. “Okay, work with Hispanic kids.” It does. “Okay, but it won’t work with POOR Hispanic but it won’t work with POOR Hispanic kids.” It does. “Okay, but it won’t work in kids.” It does. “Okay, but it won’t work in rural areas.” It does. “Okay, maybe it’s rural areas.” It does. “Okay, maybe it’s wrong to say it doesn’t work. It makes wrong to say it doesn’t work. It makes sense for a principal to try it.” sense for a principal to try it.” [Replication][Replication]

Let’s apply these ideas to medicine.Let’s apply these ideas to medicine. Let’s say your child is ill with an Let’s say your child is ill with an unusual and deadly infection. The unusual and deadly infection. The physician says, “Here’s a prescription physician says, “Here’s a prescription for Megabiotic. I and my colleagues for Megabiotic. I and my colleagues developed this drug ourselves.”developed this drug ourselves.”You ask, “Will it work, Doctor?”You ask, “Will it work, Doctor?”Physician. “Oh, yes. It will fix her Physician. “Oh, yes. It will fix her right right up.” up.”You. “Does it have bad side You. “Does it have bad side effects?”effects?”Physician. “Oh, no. No side effects.”Physician. “Oh, no. No side effects.”You. “How do you know, You. “How do you know, Doctor?”Doctor?”

““How do you know, Doctor?”How do you know, Doctor?”

Now THAT is THE question. Now THAT is THE question.

““How do you know?”How do you know?”

““What makes you so sure?”What makes you so sure?”

““Do you have trustworthy evidence Do you have trustworthy evidence to back up your claim?”to back up your claim?”

What if the physician says, “I KNOW What if the physician says, “I KNOW I’m right,” or “I’ve been in this I’m right,” or “I’ve been in this business a long time. I know what business a long time. I know what I’m talking about,” or “We I’m talking about,” or “We conducted research, and we believe conducted research, and we believe that the research tells us that that the research tells us that Megabiotic is effective.”Megabiotic is effective.”

Would you give Megabiotic to your Would you give Megabiotic to your child? Of course NOT! Why? child? Of course NOT! Why? Because the physician is too FULL Because the physician is too FULL OF HIMSELFOF HIMSELF . Too sure. He . Too sure. He presumes he’s right and expects you presumes he’s right and expects you to believe him on his SAY SO. to believe him on his SAY SO.

But what if, instead, the physician But what if, instead, the physician reached into the file cabinet and took out reached into the file cabinet and took out a dozen journal articles, and then led you a dozen journal articles, and then led you through them. The physician says, through them. The physician says, “Here’s is our research.”“Here’s is our research.”

• Megabiotic was tested in Megabiotic was tested in 10 10 experimental experimental studiesstudies. [Good. They DID research. And . [Good. They DID research. And they they REPLICATED REPLICATED the research to see if the research to see if findings were reliable, dependable.]findings were reliable, dependable.]

• Each study Each study lasted from one to five yearslasted from one to five years. . [Good. It was [Good. It was longitudinal longitudinal research.]research.]

• Each study had at least Each study had at least 1000 persons1000 persons in in it. it. [Good. They used large samples so they [Good. They used large samples so they could tell if the drug had different effects could tell if the drug had different effects on different groups of persons.]on different groups of persons.]

• Each study sample had a mix of Each study sample had a mix of different kinds different kinds of personsof persons in it: younger/older, more ill/less ill, in it: younger/older, more ill/less ill, male/female. [Good. The samples were male/female. [Good. The samples were representativerepresentative of the population of persons of the population of persons who might TAKE the medication. Therefore, who might TAKE the medication. Therefore, they can GENERALIZE their FINDINGS from they can GENERALIZE their FINDINGS from the samples TO the larger population.]the samples TO the larger population.]

• Each study compared the effects of Megabiotic Each study compared the effects of Megabiotic with with otherother medications ( medications (alternative alternative treatmentstreatments) and with ) and with no no treatments (control treatments (control group). group). [They used [They used COMPARISON COMPARISON groups. Good. groups. Good. Therefore, they can tell if the Therefore, they can tell if the medication medication works better than other medications and works better than other medications and works better than taking no medications. Why works better than taking no medications. Why take Megabiotic if it doesn’t work as well as take Megabiotic if it doesn’t work as well as taking NOTHING or taking something else?]taking NOTHING or taking something else?]

• Each study measured the Each study measured the exact amountexact amount of of infection before treatment, each week during infection before treatment, each week during treatment, and at the end of the course of treatment, and at the end of the course of treatment. The measurement was objective---treatment. The measurement was objective---the amount of infection in the blood. the amount of infection in the blood. [Good. They had [Good. They had CLEAR DEFINTIONSCLEAR DEFINTIONS of what of what Megabiotic was supposed to Megabiotic was supposed to DO---its effects. DO---its effects. (The amount of infection in the blood.) (The amount of infection in the blood.) Therefore, they could measure its effects. They Therefore, they could measure its effects. They also had also had PRECISE PRECISE and and OBJECTIVE OBJECTIVE measures. measures. Therefore, they could tell exactly Therefore, they could tell exactly HOW MUCHHOW MUCH of an effect it had---of an effect it had---quantitative dataquantitative data.].]

• Side effectsSide effects were also measured, such as blood were also measured, such as blood pressure, skin rash, and kidney function. pressure, skin rash, and kidney function. [Good. They were HONEST. If you only look [Good. They were HONEST. If you only look for GOOD effects, you can always find for GOOD effects, you can always find SOMETHING. If you don’t look for harmful SOMETHING. If you don’t look for harmful effects, it will look like the medication doesn’t effects, it will look like the medication doesn’t HAVE any.]HAVE any.]

•All of the studies found that persons All of the studies found that persons who received Megabiotic had NO who received Megabiotic had NO serious side effects. serious side effects.

[Good. These are [Good. These are PRECISEPRECISE findings.] findings.]

•Persons who received Megabiotic Persons who received Megabiotic showed showed directly-observabledirectly-observable (in the (in the blood) reductions in infection each blood) reductions in infection each week until, by the end of one year, week until, by the end of one year, 95% of the persons were cured. 95% of the persons were cured.

[Good. More precise findings---in [Good. More precise findings---in contrast to contrast to qualitativequalitative data, such as, data, such as, “Persons who took Megabiotic “Persons who took Megabiotic said said they feltthey felt better.” What does that better.” What does that mean?]mean?]

•The research was not conducted by The research was not conducted by the drug company, but by the drug company, but by independent researchers who had independent researchers who had nothing to gain whether Megabiotic nothing to gain whether Megabiotic was effective or not effective. was effective or not effective. [Good. The research was NOT likely [Good. The research was NOT likely to be rigged to get good results that to be rigged to get good results that would make the medication look would make the medication look good----for example, dropping from good----for example, dropping from the studies any persons who were the studies any persons who were NOT making progress. Of course NOT making progress. Of course the medication would look effective the medication would look effective if you don’t count all of the persons if you don’t count all of the persons with whom it was NOT effective!]with whom it was NOT effective!]

Now would you let your child Now would you let your child take the medication?take the medication?

Probably.Probably.

Why?Why?

Why is the research report from Why is the research report from your physician convincing? your physician convincing?

Let’s check the Baloney Detector.Let’s check the Baloney Detector.

1.1. Was the language flowery and vague? Was the language flowery and vague? Did it appeal to your feelings and not to Did it appeal to your feelings and not to your intelligence? [What do you your intelligence? [What do you think…………..]think…………..]

2. Were the claims too good to be true; 2. Were the claims too good to be true; exaggeratedexaggerated; puffed up? Or do they ; puffed up? Or do they soundsound reasonable reasonable? [What do you ? [What do you think?.................]think?.................]

3.3. Are the claims about EFFECTS clear? Are the claims about EFFECTS clear? Do you know WHAT they are talking Do you know WHAT they are talking about---what the EFFECTS are supposed about---what the EFFECTS are supposed to be; what they are MEASURING? [What to be; what they are MEASURING? [What was the variable that had to do with was the variable that had to do with effects? It had something to do with the effects? It had something to do with the blood…......]blood…......]This criterion has to do with theThis criterion has to do with the definitiondefinition of variables of variables..

4.4. Did they do any research that Did they do any research that tested the USEtested the USE of the medication of the medication in the real world (this is called in the real world (this is called empirical researchempirical research)? [What do )? [What do you think?................. ]you think?................. ]

5.5. If they DID research, did they If they DID research, did they study a study a LARGE enough sampleLARGE enough sample of of persons (not just 20, in a persons (not just 20, in a laboratory) so that they can tell laboratory) so that they can tell HOW OFTEN it worked very well, a HOW OFTEN it worked very well, a little bit, not at all, or was little bit, not at all, or was harmful? [What do you harmful? [What do you think?.................]think?.................]

6.6. If they DID research, did they study If they DID research, did they study a sample of persons who were different a sample of persons who were different in MANY ways (younger and older, male in MANY ways (younger and older, male and female, more sick and less sick), so and female, more sick and less sick), so that the sample was like the LARGER that the sample was like the LARGER population that might USE Megabiotic? population that might USE Megabiotic? [This is called using a [This is called using a representative representative samplesample.] [What do you think?.................].] [What do you think?.................]

7. If they DID research, did they do it 7. If they DID research, did they do it again and again (this is called again and again (this is called replicationreplication), to see if the effects were ), to see if the effects were RELIBALE RELIBALE (dependable, can be counted (dependable, can be counted on)? If not, maybe any good results were on)? If not, maybe any good results were a fluke. [What do you think?.................]a fluke. [What do you think?.................]

8.8. If they did research, did they If they did research, did they do it over a long period (this is do it over a long period (this is called called longitudinal researchlongitudinal research) to ) to see the long term effects? Maybe see the long term effects? Maybe Megabiotic only works for a day Megabiotic only works for a day or two. [What do you or two. [What do you think?.................]think?.................]

9.9. The research was conducted The research was conducted by by independent researchersindependent researchers who who had no stake in the outcomes.had no stake in the outcomes.

In other words, the research report is In other words, the research report is convincing because: convincing because:

•The claims about effects are The claims about effects are believable.believable.

•The report gives you solid REASONS The report gives you solid REASONS to believe that the medication is likely to believe that the medication is likely to cure your child’s infection and not to cure your child’s infection and not cause harm.cause harm.

•The report gives you evidence that is The report gives you evidence that is hard to dismiss as just baloney.hard to dismiss as just baloney.

•You can’t say, “It won’t work with my You can’t say, “It won’t work with my child,” because the research on child,” because the research on Megabiotic was tested with thousands Megabiotic was tested with thousands of persons---many of whom were just of persons---many of whom were just like your child.like your child.

•You can’t say, “Maybe persons got well by You can’t say, “Maybe persons got well by themselves. It wasn’t Megabiotic,” themselves. It wasn’t Megabiotic,” because persons in the control group because persons in the control group (who got no Megabiotic) did NOT get (who got no Megabiotic) did NOT get well.well.

•You can’t say, “They didn’t measure You can’t say, “They didn’t measure improvement,” because in fact they improvement,” because in fact they measured what can be SEEN (the amount measured what can be SEEN (the amount of infection in the blood)---an of infection in the blood)---an objective objective measuremeasure of improvement, not an opinion. of improvement, not an opinion.

•You can’t say, “The effects are probably You can’t say, “The effects are probably temporary; persons will get sick again,” temporary; persons will get sick again,” because some of the studies lasted five because some of the studies lasted five years, and no one got sick again with the years, and no one got sick again with the infection.infection.

•You can’t say, “The research was You can’t say, “The research was rigged to get good results. For rigged to get good results. For example, researchers dropped example, researchers dropped from the study all persons who from the study all persons who were NOT getting well.” You were NOT getting well.” You can’t say this because the drug can’t say this because the drug company (that wanted the drug company (that wanted the drug to work) did not DO the research.to work) did not DO the research.

Now, let’s apply the same reasoning to Now, let’s apply the same reasoning to education.education.

You are a teacher.You are a teacher.

Or a curriculum specialist for a school or Or a curriculum specialist for a school or district.district.

Or a school principal.Or a school principal.

Or a curriculum director for a whole Or a curriculum director for a whole state….state….

You know that you must raise the You know that you must raise the reading achievement of all students, and reading achievement of all students, and especially of minority and poor students.especially of minority and poor students.

2005 NAEP Grade 4 Reading2005 NAEP Grade 4 ReadingAll Students, NationAll Students, Nation

Source: National Center for Education Statistics, NAEP Data Explorer, http://nces.ed.gov/nationsreportcard/nde/

38

33

30

0%

20%

40%

60%

80%

100%

All Students

Per

cen

t o

f S

tud

ents

Proficient/Advanced

Basic

Below Basic

Source: National Center for Education Statistics, NAEP Data Explorer, http://nces.ed.gov/nationsreportcard/nde

59

28

56 51

25

29

32

2930

35

13

40

15 18

40

0%

20%

40%

60%

80%

100%

AfricanAmerican

Asian Latino NativeAmerican

White

Per

cent

of S

tude

nts

Proficient/Advanced

Basic

Below Basic

2005 NAEP Grade 4 Reading by Race/Ethnicity, Nation

Are you going to select curriculum Are you going to select curriculum materials based on a clever salespitch?materials based on a clever salespitch?

Use 21st Century Literacy! Use 21st Century Literacy! Prepare your students for life-long Prepare your students for life-long

literacy in a global society.literacy in a global society.Our materials are child-centered, Our materials are child-centered,

developmentally appropriate, developmentally appropriate, holistic, authentic, and teacher-holistic, authentic, and teacher-

friendly.friendly.21st Century Literacy ensures that 21st Century Literacy ensures that

ALL of your students will become ALL of your students will become proficient in ALL reading skills.proficient in ALL reading skills.

You say, “I wonder if 21st You say, “I wonder if 21st Century Literacy can repair my Century Literacy can repair my carcar, too. ALL students? , too. ALL students?

Oh, sure! Let’s see the Oh, sure! Let’s see the datadata.”.”

Skeptical! Skeptical!

Good. Good.

You are protecting your students.You are protecting your students.

You write to the publisher, or you go to the You write to the publisher, or you go to the publisher’s website and you open the link that publisher’s website and you open the link that says, “Research.” What do you find?says, “Research.” What do you find?

You find “publications” that are:You find “publications” that are:

1. Testimonials.1. Testimonials.““I’m a second grade teacher at Oaken I’m a second grade teacher at Oaken Bucket Elementary. We’ve used 21st Bucket Elementary. We’ve used 21st Century Literacy for three years. We all Century Literacy for three years. We all love it! Our children are so proud of love it! Our children are so proud of their skills. And the authentic pictures their skills. And the authentic pictures help the children to guess what the help the children to guess what the words say.” words say.”

You say, “O, terrific! How about some You say, “O, terrific! How about some datadata? ? Can your kids Can your kids READREAD?!”?!”

2. Reports of very weak research.2. Reports of very weak research. For For example, a school used 21st Century example, a school used 21st Century Literacy, but:Literacy, but:

a.a. Did NOT measure students’ reading Did NOT measure students’ reading proficiency proficiency before before 21st Century Literacy 21st Century Literacy was was introduced. So, they only have post-introduced. So, they only have post-test (end test (end of year) scores. Therefore, of year) scores. Therefore, it’s impossible to it’s impossible to tell tell how much the how much the students students learnedlearned, if , if anything.anything.

b.b. Did not use Did not use standardized and standardized and validated validated instrumentsinstruments to measure to measure reading reading proficiency. Therefore, it’s proficiency. Therefore, it’s impossible to tell if impossible to tell if the instruments the instruments really measured reading.really measured reading.

c. Did not measure c. Did not measure all five mainall five main reading skills--- reading skills---

phonemic awareness, phonics, fluency, phonemic awareness, phonics, fluency, vocabulary, comprehension. Therefore, it’s vocabulary, comprehension. Therefore, it’s impossible to tell if all reading skills impossible to tell if all reading skills improved.improved.

d.d. Did not use an Did not use an equivalent control groupequivalent control group that that received a different reading program. received a different reading program. Therefore, it’s impossible to tell if 21st Therefore, it’s impossible to tell if 21st Century Literacy Century Literacy did any betterdid any better than the usual than the usual (ineffective) materials. It’s also impossible to (ineffective) materials. It’s also impossible to rule out the possibilityrule out the possibility that students’ reading that students’ reading proficiency proficiency at the end of the year was NOT at the end of the year was NOT due to 21st Century Literacy, but was due to due to 21st Century Literacy, but was due to “extraneous variables”“extraneous variables” such as assistance such as assistance reading at home, or maturation.reading at home, or maturation.

e.e. Did not Did not check the reliabilitycheck the reliability of of observers who tested the students. observers who tested the students. Therefore, it’s impossible to tell of the Therefore, it’s impossible to tell of the data are data are accurateaccurate..

f.f. Did not continue testing students’ reading Did not continue testing students’ reading proficiency (in a proficiency (in a longitudinallongitudinal way). way). Therefore, it’s impossible to tell if any Therefore, it’s impossible to tell if any effects of 21st Century Literacy are long effects of 21st Century Literacy are long lasting, or fade away in a few months.lasting, or fade away in a few months.

g. g. Did not replicateDid not replicate the “research” with the “research” with different samples (classes, schools). different samples (classes, schools). Therefore, it’s impossible to tell if 21st Therefore, it’s impossible to tell if 21st Century Literacy would work anywhere Century Literacy would work anywhere else.else.

And yet, the website claims,And yet, the website claims,““Our scientific research on 21st Century Our scientific research on 21st Century Literacy shows that it is highly effective.” Literacy shows that it is highly effective.”

Is there any reason to believe this?Is there any reason to believe this?

No. But many thousands of teachers, No. But many thousands of teachers, curriculum specialists, principals, and curriculum specialists, principals, and states will “adopt” 21st Century Literacy states will “adopt” 21st Century Literacy because because they can’t tell the difference they can’t tell the difference between baloney and serious claimsbetween baloney and serious claims based on rigorous scientific research. based on rigorous scientific research.

And the children will pay for this And the children will pay for this mistake!mistake!

But let’s imagine that ONE DAY But let’s imagine that ONE DAY educational research on what educational research on what works (works (evaluation research at evaluation research at Level 3)Level 3) is conducted the way is conducted the way medical research is conducted. medical research is conducted.

Or even the way research on Or even the way research on toothpaste and dental floss is toothpaste and dental floss is conducted. conducted.

A newly develop reading curriculum---A newly develop reading curriculum---Classical Reading---is tested this way.Classical Reading---is tested this way.

1.1. Classical Reading was tested in 10 Classical Reading was tested in 10 experimental studies. [experimental studies. [REPLICATION REPLICATION shows if findings are reliable, shows if findings are reliable, dependable.]dependable.]

2.2. Researchers obtained Researchers obtained samples samples from different areas of the countyfrom different areas of the county and and from different locations within states. from different locations within states. Therefore, overall, the samples were Therefore, overall, the samples were representative of the populationrepresentative of the population of of students, schools, and teachers who students, schools, and teachers who might use Classical Reading.might use Classical Reading.

3.3. Each study lasted from one to two Each study lasted from one to two years. [years. [Longitudinal researchLongitudinal research shows shows whether any effects are whether any effects are DURABLEDURABLE. Why . Why use a curriculum if you don’t know whether use a curriculum if you don’t know whether in two years students will be no more in two years students will be no more skilled than students who did NOT receive skilled than students who did NOT receive the new curriculum?]the new curriculum?]

4.4. Each study had at Each study had at least 100 studentsleast 100 students in it. in it. [These [These samples are large enoughsamples are large enough that they that they are likely to have diversity on important are likely to have diversity on important characteristics, such as sex, social class, characteristics, such as sex, social class, and assistance fro family. This way, you and assistance fro family. This way, you can can DISAGGREGATEDISAGGREGATE the data to see if the data to see if Classical Reading is more or less effective Classical Reading is more or less effective with certain with certain SUBSUBgroups.] groups.]

5.5. Each study Each study comparedcompared the the effects of Classical Reading with effects of Classical Reading with otherother (competing new) reading (competing new) reading materials (alternative treatments) materials (alternative treatments) and with and with no no treatments (control treatments (control group that received the usual group that received the usual reading materials). reading materials).

[By using [By using COMPARISONCOMPARISON groups, groups, you can tell if Classical Reading you can tell if Classical Reading works better than other works better than other programs.]programs.]

6.6. Researchers used Researchers used random random allocationallocation to create the comparison to create the comparison groups. [This makes it likely that groups. [This makes it likely that the groups were the groups were equivalent on equivalent on important factorsimportant factors such as ethnicity, such as ethnicity, social class, and sex. Therefore, if social class, and sex. Therefore, if the comparison groups are nearly the comparison groups are nearly the same in their characteristics, the same in their characteristics, and the only obvious difference is and the only obvious difference is the reading program each group the reading program each group received, then, logically, received, then, logically, differences differences in reading achievement (at the post-in reading achievement (at the post-test) are likely to be the result of test) are likely to be the result of the reading programthe reading program.].]

7.7. Each study measured the Each study measured the exact exact amountamount of reading proficiency in all five of reading proficiency in all five main reading skills main reading skills beforebefore reading reading instruction began (at the start of the instruction began (at the start of the year), each week year), each week during during school, and at the school, and at the endend of the year. [This way, researchers of the year. [This way, researchers could tell if there was a could tell if there was a TREND TREND in in learning, and could compare earlier and learning, and could compare earlier and later scores to see how much change there later scores to see how much change there was.]was.]

8.8. The measurement was The measurement was directdirect; reading ; reading proficiency was measured by measuring proficiency was measured by measuring READING behavior, not some READING behavior, not some indirect indirect substitutesubstitute, such as how much students , such as how much students enjoyed reading. enjoyed reading.

9.9. Measurement was Measurement was objective and objective and quantitativequantitative (not subjective (not subjective impressions); the behavior could be impressions); the behavior could be observed by observed by many personsmany persons; for example, ; for example, the rate of words read correctly per the rate of words read correctly per minute (fluency), or the number of minute (fluency), or the number of comprehension questions answered comprehension questions answered correctly. correctly. [This means the researchers could tell [This means the researchers could tell exactly exactly HOW MUCHHOW MUCH of an effect, if any, of an effect, if any, the reading programs had. the reading programs had.

10.The researchers used 10.The researchers used standardized standardized instrumentsinstruments that were known to be that were known to be validvalid—to measure what they purport to —to measure what they purport to measure and to yield accurate findings.measure and to yield accurate findings.

11.The researchers measured 11.The researchers measured some of the same reading skills some of the same reading skills (e.g., decoding words) in several (e.g., decoding words) in several different way---different way---triangulationtriangulation. . [You can have greater confidence [You can have greater confidence if findings from different if findings from different measurements say much the measurements say much the same thing.]same thing.]

12.The researchers assessed the 12.The researchers assessed the reliabilityreliability of testers’ data to of testers’ data to ensure accuracy.ensure accuracy.

13.Researchers also 13.Researchers also interviewedinterviewed teachers and students teachers and students ((qualitative dataqualitative data) to see if there ) to see if there were any negative “side effects.” were any negative “side effects.” For instance, even if Classical For instance, even if Classical Reading was effective, teachers Reading was effective, teachers might feel that it is too hard to might feel that it is too hard to use properly. Therefore, they use properly. Therefore, they would probably not use it very would probably not use it very long. long.

14.The researchers found differences in 14.The researchers found differences in student achievement from study to student achievement from study to study, but in general, study, but in general, 9 out of 10 9 out of 10 studiesstudies showed that students who showed that students who received Classical Reading achieved received Classical Reading achieved more in more in all five reading skillsall five reading skills by the by the end of the year than students who end of the year than students who received other programs. And most of received other programs. And most of the differences were the differences were so large that so large that statistical tests showed that these statistical tests showed that these differences were “statistically differences were “statistically significant”significant”---not likely the result of ---not likely the result of chance. [Such consistent and chance. [Such consistent and statistically significant differences statistically significant differences make any claims about effectiveness make any claims about effectiveness pretty strongpretty strong.].]

15.Most of the research was conducted by 15.Most of the research was conducted by independent researchersindependent researchers who had who had nothing to gain whether Classical nothing to gain whether Classical Reading was effective or not effective. Reading was effective or not effective.

[Therefore, the research was [Therefore, the research was NOT likely NOT likely to be riggedto be rigged to get good results that to get good results that would make the Classical Reading look would make the Classical Reading look good----for example, dropping from the good----for example, dropping from the studies any students who were NOT studies any students who were NOT making progress. Of course Classical making progress. Of course Classical Reading would look effective if Reading would look effective if researchers did not use the data on researchers did not use the data on students with whom it was NOT students with whom it was NOT effective!]effective!]

Are you inclined to believe Are you inclined to believe them? them?

Would you try Classical Would you try Classical Reading?Reading?