let the buyer beware - great lakes centertimely fashion, to write reviews of privately produced...

http://www.greatlakescenter.org

LET THE BUYER BEWARE

AN ANALYSIS OF THE SOCIAL SCIENCE VALUE AND METHODOLOGICAL QUALITY OF EDUCATIONAL STUDIES PUBLISHED BY THE

MACKINAC CENTER FOR PUBLIC POLICY (1990-2001)

By Peter W. Cookson, Jr., Ph.D. Teachers College, Columbia University

Alex Molnar, Ph.D.

College of Education, Arizona State University

Katie Embree, Ed.D. Teachers College, Columbia University

E

The Great Lakes Center for Education Research & Practice PO Box 1263

East Lansing, MI 48826 Email: [email protected]

Web Site: http://www.greatlakescenter.org

September 2001


This research was supported by a grant from the Great Lakes Center for Education Research and Practice. The authors wish to thank the Mackinac Center for Public Policy for its cooperation in providing us with the materials used in this analysis.


Table of Contents Executive Summary ................................................................................................................ 1

Findings................................................................................................................................ 2 Implications and Recommendations ................................................................................. 2

What is the Mackinac Center? .............................................................................................. 4 The Purpose of the Study ....................................................................................................... 7 Methodology ............................................................................................................................ 9 Description of the Studies Analyzed.................................................................................... 12 Evaluation Instrument ......................................................................................................... 14 Findings.................................................................................................................................. 15 Implications ........................................................................................................................... 30 Recommendations ................................................................................................................. 32 Appendix A -- Mackinac Center Policy Staff......................................................................... 33 Appendix B -- Mackinac Center Board of Directors .............................................................. 34 Appendix C -- Mackinac Center Board of Scholars ............................................................... 35 Appendix D -- Mackinac Center Descriptive Overview of Opinion Essays .......................... 36 Appendix E -- Descriptive Overview of Administrative and Legislative Guides .................. 38 Notes ....................................................................................................................................... 39 About the Authors................................................................................................................. 41

of 41


Executive Summary

The Michigan-based Mackinac Center for Public Policy has emerged as one of the largest and most prolific state-based think tanks in the United States, with a mission of enabling “Michigan citizens and other decision-makers to better evaluate Michigan public policy options … from a ‘free market’ perspective.” Notwithstanding its description of itself as “non-partisan,” the organization supports market solutions for public policy challenges and opposes government intervention – positions that can be fairly characterized as politically conservative. The Mackinac Center states it is committed to delivering “the highest quality and most reliable research on Michigan issues.” This report puts that claim to the test. An independent review of the Center’s documents is important because private think tank research often enters the mainstream of public discourse without being rigorously scrutinized. The authors of this report obtained a list of all Mackinac Center publications on the subject of education from 1990 to May 2001 from the Center’s web-site. The 22 documents described as “studies” were sorted into four categories: Original research, interpretive research, opinion essay, and administrative and legislative guide. Only Original research – the collection of empirical data using social science methodology – and Interpretive research – primarily secondary analyses of research conducted by others – were included in this evaluation. The focus of this evaluation is the social scientific quality of these 14 documents. Seven studies were classified as original research and seven as interpretive research. They were evaluated according to established guidelines for social science research. These guidelines are that research be original and important; that it use instruments demonstrated to be reliable and valid; that its outcome measures clearly relate to the variables studied; that the research design fully and unambiguously tests the hypothesis; and that test participants are representative of the population to which generalizations are made. To rate the quality of the studies, evaluators developed evaluation instruments based on the applicable guidelines. A score of 3 points indicated that a study was judged to have achieved the standard of quality necessary for publication in a peer-reviewed social science research journal. A score of 2 points indicated that the study, in general, adequately met social science research standards. A score of 1 point indicated that a study did not adequately meet social science research standards. An average was then taken of scores on each of the applicable guidelines to derive an overall score for each study. The overall scores for Mackinac Center studies ranged from 1.14 to 3.0.

of 41


Findings The evaluation of education related documents described by the Mackinac Center as studies found that:

• Many of the documents described by the Mackinac Center as “studies” do not represent genuine social science research.

• Overall, the quality of education studies published by the Mackinac Center that can

be considered social science research ranges from inadequate to just adequate. Applying the rating system established in this evaluation, studies classified as original research received, on average, an overall quality score of 1.80 on a scale of 1 to 3, with 3 being the highest. Studies classified as interpretive research received, on average, an overall quality score of 2.09.

• Few Mackinac Center sponsored studies are of high quality and very few would be

accepted for publication in a peer-reviewed academic journal. Of the 14 studies classified as original or interpretive research, only one was unambiguously of high enough quality that it would be considered for publication in a peer-reviewed academic journal.

Implications and Recommendations Mackinac Center research is often of low quality and because of this it should be treated with considerable skepticism by the public, policy makers and political leaders. Indeed, much of the work of the Mackinac Center may have caused more confusion than clarity in the public discussion of the issues that it has addressed by systematically ignoring evidence that does not agree with its proposed solutions. To better serve the public, a process should be established that allows the public, policy makers, and the media to become more discriminating consumers of research, so that political beliefs cannot so easily disguise themselves as social science. To that end, the Michigan legislature should work with education research organizations and provide funds to:

1. Establish a network of education experts with diverse expertise who are willing, in a timely fashion, to write reviews of privately produced documents and to allow those reviews to be distributed to policy makers and the public;

2. Create and maintain a system for monitoring the work produced by private think tanks to facilitate the timely review of reports they issue; and

3. Support a web-site containing a database indicating the social science reliability of the hundreds of studies being published by think tanks and policy centers. This web-site would allow legislators to assess the quality of the research that is being used to urge them to take a particular course of action.

of 41


For the citizens of Michigan it is important to understand that research supported by the Mackinac Center is often inadequate as a basis for formulating education policy.

of 41


What is the Mackinac Center? The Mackinac Center for Public Policy, located in Midland, Michigan was founded in 1988. The Center’s stated mission is to “equip Michigan citizens and other decision-makers to better evaluate Michigan public policy options and to do so from a ‘free market’ perspective.1 Since its founding, the Mackinac Center has developed into one of the largest and most prolific state-based think tanks in the United States. Its reports and other publications are frequently cited by Michigan elected officials. Its President, Lawrence W. Reed, a former college economics professor, has alone authored more than 800 newspaper columns and articles in the last ten years. Since 1994 Reed has also served on the Board of Trustees of the Foundation for Economic Education. 2 The Mackinac Center describes itself as follows:

The Mackinac Center for Public Policy is a nonpartisan research and educational organization devoted to improving the quality of life for all Michigan citizens by promoting sound solutions to state and local policy questions. The Mackinac Center assists policy makers, scholars, business people, the media and the public by providing objective analysis of Michigan issues. The goal of all Center reports, commentaries and educational programs is to equip Michigan citizens and other decision-makers to better evaluate policy options. The Mackinac Center for Public Policy is broadening the debate on issues that has for many years been dominated by the belief that government intervention should be the standard solution. Center publications and programs, in contrast, offer an integrated and comprehensive approach that considers:

All Institutions. The Center examines the important role of voluntary associations, business, community and family, as well as government.

All People. Mackinac Center research recognizes the diversity of Michigan citizens and treats them as individuals with unique backgrounds, circumstances and goals.

All Disciplines. Center research incorporates the best understanding of economics, science, law, psychology, history and morality, moving beyond mechanical cost/benefit analysis.

All Times. Center research evaluates long-term consequences, not simply short-term impact.

Committed to its independence, the Mackinac Center for Public Policy neither seeks nor accepts any government funding. It enjoys the support of foundations, individuals, and businesses who share a concern for Michigan's future and recognize the important role of sound ideas. The Center is a nonprofit, tax-exempt organization under Section 501(c)(3) of the Internal Revenue Code. Copyright © 2001 Mackinac Center for Public Policy. 3

of 41


The Center focuses on economic policy and avoids what it considers to be social issues such as abortion and gun control because, according to the Center’s web-site, the Center believes that a grasp of sound economic principles will lead to a “more sophisticated level of political and economic understanding among Michigan citizens and decision-makers.” 4 Despite its focus on economic policy, since 1990, the Mackinac Center has published numerous education related documents. This evaluation reviews education related publications described by the Center as studies. While the Mackinac Center’s literature states that is non-partisan, it is clear that it has a political point-of-view and an ideological orientation that favors market solutions to public policy challenges. As its mission statement indicates, it considers itself to be in opposition to those who believe that government intervention is an important mechanism for improving the quality of life and quality of education in Michigan. The Center’s point-of-view is well expressed by the following quote:

Modern economic experience demonstrates overwhelmingly that the free market is a powerful engine of economic prosperity, and nations the world over are clamoring to shed the chains of central planning and unleash creative energy of free men and women. The principles of the American Revolution – individual liberty, limited government, the free market, and the rule of law – have become the dominant paradigm of enlightened society. 5

The Center’s mission statement concludes as follows:

We look forward to the day when the myths and fears of free market capitalism are dispelled, along with the misplaced faith in a benevolent, omnipotent state. By focusing on the actual problems and understanding the proper role of public and private institutions, we can give all Michigan citizens the greatest opportunity for peace, prosperity and freedom. 6

Thus, although the Mackinac Center does not refer to itself as conservative, it is safe to say that the Center’s belief in market solutions to educational and other public policy challenges places it on what is conventionally thought of as the Right end of the political spectrum. This general worldview results in policy positions that favor privatization, marketization and commercialization as desirable methods of deregulation. Not surprisingly, therefore, much of the work of the Mackinac Center provides analyses associated with some form of deregulation. Including the Center’s President, the staff of the Mackinac Center consists of fourteen people, Appendix A, and is governed by a Board of Directors whose names appear in Appendix B. The Center is supported intellectually by a Board of Scholars whose names appear in Appendix C. In addition, Mackinac lists over 100 authors whose names are available on the Mackinac web-site. 7

of 41


The Mackinac Center is an important public policy organization and therefore must be taken seriously in the ongoing debate over education policy in Michigan and, by implication, by the rest of the United States. For this reason the quality of the educational research sponsored and published by the Mackinac Center should be carefully considered and the quality of its research analyzed.

of 41


The Purpose of the Study This evaluation assesses the quality of the educational research conducted by the Mackinac Center by reviewing the standard of scholarship of the studies it releases. The Mackinac Center offers the following assurance about the quality of its scholarship:

The Mackinac Center for Public Policy is committed to delivering the highest quality and most reliable research on Michigan issues. The Center guarantees that all original factual data are true and correct and that information attributed to other sources is accurately represented.

The Center encourages rigorous critique of its research. If the accuracy of any material fact or reference to an independent source is questioned and brought to the Center’s attention with supporting evidence, the Center will respond in writing. If an error exists, it will be noted in an errata sheet and will accompany all subsequent distribution of the publication, which constitutes the complete and final remedy under this guarantee. 8

An independent review of the Center’s documents is important because the research of private think tanks often enters the mainstream of public discourse without being rigorously scrutinized. Most media, for example, do not have the time, capacity, or the inclination to dig beneath the surface to determine the validity of research results and too often may place the reports of these think tanks on par with more rigorous academic studies that are published in academic journals. One of the hallmarks of a high quality academic journal is, for example, the strict adherence to the principle of “blind” review. A blind review is one where the reviewer does not know the name(s) of the researchers or the researchers’ institutional affiliations. This ensures that quality is not compromised by favoritism or institutional loyalty. This review process is rigorous; a rejection rate of ninety percent is common. Blind reviews go through several revisions; it is unheard of that an article would be accepted for publication without edits and revisions. This form of peer-review relies on the absolute integrity of the process. Editors of scientific journals are chosen for their reputations as scholars and violations of the process are a matter of great concern. By utilizing this process, social scientists can have confidence in the research published, whether or not they find the conclusions personally to their liking. As Molnar has pointed out concerning the publication of non peer-reviewed reports, “amid the welter of such reports and the media coverage of them, there seem to be few guideposts for readers to measure the quality of the information that they are receiving.” 9 The danger posed by widely publicized non peer-reviewed reports of questionable social science value has been the subject of considerable discussion. Several recent articles and essays have addressed the issue. 10

of 41


This study is intended to serve as a tool for policy makers and the people of Michigan as they consider the merits of Mackinac Center documents. To accomplish this purpose, the study addresses the following research questions:

1. What are the major educational research concerns of the Mackinac Center? 2. Who are the Mackinac researchers and authors?

3. What is the process of review before publication?

4. What is the quality of the Mackinac Center’s original research?

5. What is the quality of the Mackinac Center’s interpretive research?

6. What is the overall quality of the educational research published by the Mackinac

Center?

of 41


Methodology In order to assess the social scientific quality of the educational studies published by the Mackinac Center, the following process was employed.

• A complete list of all Mackinac Center education related publications from 1990 to May, 2001 was obtained by reviewing the Mackinac Center’s web-site. 11

• The Mackinac Center publishes a wide variety of educational documents, including

studies, newspaper articles, and commentaries. Only those publications defined by the Mackinac Center as studies were included in this research.

• The twenty-two documents described by the Mackinac Center as studies were

organized into four categories: original research, interpretive research, opinion essay and administrative and legislative guides. The categories are defined below:

1. Original research: the collection of empirical data using social science

methodology. 2. Interpretive research: primarily secondary analyses of research conducted by

others. 3. Opinion essay: an essay which argues from a point-of-view and marshals

evidence to support that point-of-view much like a newspaper editorial. 4. Administrative and legislative guide: a publication intended to assist legislators in

drafting legislation or to be used by administrators in establishing rules or regulations.

• Only those documents classified as original and interpretive research were included in

this evaluation because the focus here is the social scientific quality of the research published by the Mackinac Center not the quality of the rhetoric or logic expressed in an opinion essay or an administrative and legislative guide. Seven documents were classified as original research. Seven studies were classified as interpretive research.

• Guidelines for evaluating the social scientific validity of the fourteen studies were created using a checklist suggested by the American Psychological Association (APA) in their publication manual. This checklist is based on research by K. M. Bartol, “Survey Results from Editorial Board Members: Lethal and Nonlethal Errors” (1981). 12 The American Psychological Association is one of the world’s most reliable and respected social science associations. More generally, the guidelines used in this study are the guidelines that are used when evaluating social scientific educational research. These guidelines have been turned into essential questions as can be seen in Figure 1 below.

of 41


Figure 1 – Essential Questions for Evaluating Social Science Research 1. Is the research question significant, and is the work original and important? 2. Have the instruments been demonstrated to have satisfactory reliability and validity? 3. Are the outcome measures clearly related to the variables with which the investigation is concerned? 4. Does the research design fully and unambiguously test the hypothesis? 5. Are the participants representative of the population to which the generalizations are made? 6. Did the researcher observe ethical standards in the treatment of participants? 7. Is the research at an advanced enough stage to make the publication of results meaningful?

• The evaluation instruments are based on six of the seven questions indicated

above. The Mackinac Center has no statement of policy governing the treatment of research subjects. Moreover, none of the studies reviewed addressed this topic. University-based research involving human subjects is carefully structured to address this standard. Rather than automatically assign all Mackinac Center studies a score of one on this guideline and thus depress their ratings, it was decided not to include question number 6 (the ethical treatment of participants) when evaluating Mackinac Center research.

• A further question was added: Is the literature review thorough and balanced?

• Based on these guidelines, two evaluation instruments were created (see Figures 4

and 5 on page 17). The instrument in Figure 4 was used to evaluate documents classified as original research. The instrument in Figure 5 was used to evaluate documents classified as interpretive research.

• For the purpose of this study it was hypothesized that those documents designated

by the Mackinac Center as studies were of high enough quality to be published in a peer-reviewed academic journal. Thus, Mackinac Center documents were evaluated in a manner consistent with the way in which social science research studies are evaluated when submitted for publication. The method of evaluation was to look for evidence that the Mackinac Center’s research protocols conform to established social scientific guidelines. For example, when asking “Are the participants representative of the population to which generalizations are made?” the evaluators first assessed if the Mackinac Center clearly established the nature of the population under study (ie. individuals, groups, organizations, record data) and then whether or not a proper sample was drawn from the population (ie. random sample, stratified random sample) or whether the Mackinac Center drew samples of convenience, which are not reliably representative of the population.

The researchers operationalized each of the questions as follows: Question 1: Is the research question significant, and is the work original and important? Is the topic important? Are the questions fairly asked? Are the questions addressed in an original manner? Are they framed in such a way as to yield valid results?

of 41


Question 2: Have the instruments been demonstrated to have satisfactory reliability and validity? Have the authors addressed the issues of reliability and validity and have they tested their instruments? Question 3: Are the outcome measures clearly related to the variables with which the investigation is concerned? Do the authors collect the appropriate data to answer their research question? Question 4: Does the research design fully and unambiguously test the hypothesis? Has the research hypothesis been clearly stated? Is it testable? Is the hypothesis stated in a manner that allows for its disconfirmation? Question 5: Are the participants representative of the population to which generalizations are made? Has the proper population been identified? Is the sample representative of the population? Is the sample systematically studied or is the evidence random and anecdotal? Question 6: Is the research at an advanced enough stage to make the publication of results meaningful? Could this document be published in an academic journal? Is it written in a clear and unambiguous manner? Are the findings original and significant enough to justify publication or is it a restatement of previously published research? Question 7: Is the literature review thorough and balanced? Have the authors reviewed all the pertinent literature or just part of the literature? Is the literature used for analytic purposes or to substantiate the authors’ point-of-view? Is the literature presented clearly and fairly enough to allow for the reader to arrive at an independent conclusion? • To establish interrater reliability, the index was used by researchers with a sample of the

fourteen Mackinac sponsored studies classified as either original or interpretive research. • After establishing interrater reliability, all fourteen studies were evaluated using this

index and each study was given a numerical score. The studies were also read more broadly for their general quality and internal consistency.

of 41


Description of the Studies Analyzed The twenty-two education related documents described as studies by the Mackinac Center are listed below. Figure 2 organizes the documents by topic. Figure 2: Twenty-Two Education Related Studies Classified by Topic School Choice Educational Choice for Michigan – September, 1991 Michigan’s Experiment with Public School Choice: A First Year Assessment – January, 1993 Teacher, Inc. A Private-Practice Option for Educators – October, 1994 The Universal Tuition Tax Credit: A Proposal to Advance Parental Choice in Education – November, 1997 School Choice in Michigan – July, 1999 The Cost of Remedial Education – August, 2000 The Impact of Limited School Choice on Public School Districts – August, 2000 Higher Education The Michigan Education Trust: A Political Economy Perspective – March, 1990 Declining Standards at Michigan Public Universities – October, 1996 School Privatization Do Private Schools Serve Difficult-to-Educate Children? – October, 1997 Unused Capacity in Privately Funded Michigan Schools – March, 1999 Public Policy Statement Modern Schools for Michigan: An Outline of Educational Reform – September, 1993 Making Schools Work: Contracting Options for Better Management – January, 1994 Doing More with Less: Competitive Contracting for School Support Services – November, 1994 Outcome-Based Education: Miracle Cure or Plague? – September, 1995 Are Michigan’s History Textbooks Reliable? - January, 1996 The Need for Debt Policy in Michigan Public Schools – March, 1998 Keeping Michigan on Track: A Blueprint for Governor Engler and the 90th Legislature – January, 1999 How Reliable are Michigan High School Economics Textbooks? – June, 1999 Teachers Unions Michigan Education Special Services Association: The MEA’s Money Machine – November, 1993 Collective Bargaining: Bringing Education to the Table – August, 1998 The Impact of School Choice on School Employee Labor Unions – June, 1999

of 41


Figure 3 organizes the twenty-two documents described by the Mackinac Center as studies by the research approach taken in each study. Figure 3 – Twenty-Two Education Related Studies Classified by Approach to Topic Original Research Michigan's Experiment with Public School Choice: A First Year Assessment – January, 1993 Declining Standards at Michigan Public Universities – October, 1996 Collective Bargaining: Bringing Education to the Table – August, 1998 Unused Capacity in Privately Funded Michigan Schools – March, 1999 The Impact of School Choice on School Employee Labor Unions – June, 1999 The Impact of Limited School Choice on Public School Districts – August, 2000 The Cost of Remedial Education – August, 2000 Interpretive Research The Michigan Education Trust: A Political Economy Perspective – March, 1990 Michigan Education Special Services Association: The MEA's Money Machine – November, 1993 Teacher, Inc. A Private-Practice Option for Educators – October, 1994 Outcome-Based Education: Miracle Cure or Plague? – September, 1995 Do Private Schools Serve Difficult-to-Educate Children? – October, 1997 The Universal Tuition Tax Credit: A Proposal to Advance Parental Choice in Education – November, 1997 School Choice in Michigan – July, 1999

Opinion Essays Educational Choice for Michigan – September, 1991 The Need for Debt Policy in Michigan Public Schools – March, 1998 Keeping Michigan on Track: A Blueprint for Governor Engler and the 90th Legislature – January, 1999 How Reliable are Michigan High School Economics Textbooks? – June, 1999 Administrative and Legislative Guide Modern Schools for Michigan: An Outline of Educational Reform – September, 1993 Making Schools Work: Contracting Options for Better Management – January, 1994 Doing More with Less: Competitive Contracting for School Support Services – November, 1994 Are Michigan's History Textbooks Reliable? – January, 1996 Of the twenty-two publications seven were classified as original research, seven as interpretive research, four as opinion essays and four as administrative and legislative guides. Only those documents classified as original and interpretive research are considered in this study.

of 41


Evaluation Instrument As discussed in the methodology section, evaluation instruments were created to analyze the quality of the educational documents classified as original research (see Figure 4 below) and as interpretive research (see Figure 5 below). Seven questions are applied to the studies classified as original research. Three are applied to the studies classified as interpretive research studies because the social scientific methodological questions are not applicable to interpretive studies. The rating system uses a one to three scale for each item. One represents the lowest rating and three the highest. A rating of one indicates that the item or the study has not adequately met the standards of acceptable social science research. A rating of two indicates that an item or study has adequately met the standards of social science research. A rating of three indicates that a particular item or the study is on par with studies published in academic journals. Figure 4 – Evaluation Instrument for Education Related Studies Classified as Original 3 – Meets standard of quality for social science research needed for publication in a peer-reviewed academic journal 2 – Adequately meets social science research standard 1 – Does not adequately meet social science research standard Study Number: Score 1. Is the research question significant, and is the work original and important?

2. Have the instruments been demonstrated to have satisfactory reliability and validity?

3. Are the outcome measures clearly related to the variables with which the investigation is concerned?

4. Does the research design fully and unambiguously test the hypothesis? 5. Are the participants representative of the population to which generalizations are made?

6. Is the research at an advanced enough stage to make the publication of results meaningful?

7. Is the literature review thorough and balanced? Total Score Figure 5 – Evaluation Instrument for Education Related Studies Classified as Interpretive 3 – Meets standard of quality for social science research needed for publication in a peer- reviewed academic journal 2 – Adequately meets social science research standard 1 – Does not adequately meet social science research standard Study Number: Score 1. Is the research question significant, and is the work original and important?

2. Is the research at an advanced enough stage to make the publication of results meaningful?

3. Is the literature review thorough and balanced? Total Score

of 41


Findings A brief description of each of the fourteen studies reviewed as provided by the Mackinac Center is below. The ratings for each study follow the description. Summaries of the opinion essays and administrative and legislative guides can be found in Appendix D and Appendix E. Mackinac Center Description and Evaluator’s Qualitative Ratings of the Seven Studies Classified as Original Research 1. Michigan’s Experiment with Public School Choice: A First Year Assessment

Adam DeVore and Robert Wittmann, January, 1993 Mackinac Center Description of Document: In 1991, the state of Michigan required each of the state’s 563 school districts to devise intra district choice plans by April 1992. Wittmann and DeVore examine those plans and conclude that what could have been a bold new beginning for school reform in Michigan turned out to be a largely unproductive extension of the status quo. Genuine choice, competition and accountability in education require far more fundamental changes that will break the monopoly of the public education establishment and create a real marketplace for education. Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 2 OUT OF 3 Question 2: Have the instruments been demonstrated to have satisfactory reliability and validity? – 1 OUT OF 3 Question 3: Are the outcome measures clearly related to the variables with which the investigation is concerned? – 2 OUT OF 3 Question 4: Does the research design fully and unambiguously test the hypothesis? – 1 OUT OF 3 Question 5: Are the participants representative of the population to which generalizations are made? – 2 OUT OF 3 Question 6: Is the research at an advanced enough stage to make the publication of results meaningful? – 1 OUT OF 3 Question 7: Is the literature review thorough and balanced? – 1 OUT OF 3 OVERALL QUALITY SCORE: 1.43 OUT OF 3

2. Declining Standards at Michigan Public Universities

Thomas F. Bertonneau, October, 1996 Mackinac Center Description of Document: Reflecting a national problem, Michigan public universities are producing graduates who are unprepared for K-12 teaching careers and the business world. The demise of the traditional core curriculum, indoctrination in the classroom, and questionable teaching methods that emphasize emotion and subjectivity over rigor and critical thinking are to blame. The study documents extensive evidence cited by employers that college graduates lack crucial communications and thinking skills, and it finds a link between poor training of aspiring teachers and declining K-12 student performance. Analysis of over 300 undergraduate course syllabi reveal the dominance of trendy, politicized course content. Assessment of Social Science Quality of Document:

of 41


Question 1: Is the research question significant, and is the work original and important? – 2 OUT OF 3 Question 2: Have the instruments been demonstrated to have satisfactory reliability and validity? - 1 OUT OF 3 Question 3: Are the outcome measures clearly related to the variables with which the investigation is concerned? - 2 OUT OF 3 Question 4: Does the research design fully and unambiguously test the hypothesis? - 1 OUT OF 3 Question 5: Are the participants representative of the population to which generalizations are made? - NA Question 6: Is the research at an advanced enough stage to make the publication of results meaningful? - 1 OUT OF 3 Question 7: Is the literature review thorough and balanced? – 1 OUT OF 3 OVERALL QUALITY SCORE: 1.33 OUT OF 3

3. Collective Bargaining: Bringing Education to the Table

LaRae G. Munk, J.D, August, 1998 Mackinac Center Description of Document: Michigan parents, citizens, and policy makers have begun an earnest discussion over the issues that affect the quality of children’s education, but one issue that is rarely considered in discussions about education reform is public school union collective bargaining. This Mackinac Center for Public Policy study is the first ever to systematically analyze the hundreds of collective bargaining agreements for every school district in a state. It examines collective bargaining’s impact on Michigan public education and makes recommendations that school boards should incorporate into their union contracts to improve their ability to direct maximum resources to the classroom and deliver quality education to students. The study also explains the historical and legal framework of public employee collective bargaining in Michigan, analyzes seven important court rulings that affect public collective bargaining issues, and advises districts on which subjects to negotiate or not negotiate into their labor contracts. Three appendices compare costs and benefits of various health care plans and present contract and financial data from the survey of Michigan’s 583 school districts. Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 3 OUT OF 3 Question 2: Have the instruments been demonstrated to have satisfactory reliability and validity? - 2 OUT OF 3 Question 3: Are the outcome measures clearly related to the variables with which the investigation is concerned? – 3 OUT OF 3 Question 4: Does the research design fully and unambiguously test the hypothesis? – 3 OUT OF 3 Question 5: Are the participants representative of the population to which generalizations are made? – 3 OUT OF 3 Question 6: Is the research at an advanced enough stage to make the publication of results meaningful? - 2 OUT OF 3 Question 7: Is the literature review thorough and balanced? – 3 OUT OF 3 OVERALL QUALITY SCORE: 2.71 OUT OF 3

4. Unused Capacity in Privately Funded Michigan Schools

Matthew J. Brouillette, March, 1999 Mackinac Center Description of Document: Many Michigan education reformers are exploring proposals to use private schools to help fix public school problems, including student overcrowding and a lack of incentives for improving student performance. The proposals, whether they involve public-to-private student transfers or expanded parental choice among all schools, depend on private schools’ willingness and ability to accommodate new students. This study, which surveyed 342 of Michigan’s 1,058 private schools, confirms that private schools have the classroom capacity

of 41


and desire to accept a significantly larger role in providing more of the state’s children with quality education. Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? - 1 OUT OF 3 Question 2: Have the instruments been demonstrated to have satisfactory reliability and validity? - 1 OUT OF 3 Question 3: Are the outcome measures clearly related to the variables with which the investigation is concerned? - 2 OUT OF 3 Question 4: Does the research design fully and unambiguously test the hypothesis? - 1 OUT OF 3 Question 5: Are the participants representative of the population to which generalizations are made? - 1 OUT OF 3 Question 6: Is the research at an advanced enough stage to make the publication of results meaningful? - 1 OUT OF 3 Question 7: Is the literature review thorough and balanced? – 1 OUT OF 3 OVERALL QUALITY SCORE: 1.14 OUT OF 3

5. The Impact of School Choice on School Employee Labor Unions

Matthew Brouillette and Jeffrey R. Williams, June, 1999 Mackinac Center Description of Document: As school choice heads for the 2000 ballot in Michigan, it is important for citizens to understand how proposals including K-12 vouchers and tuition tax credits will affect the school employee unions that exert such a powerful influence on the state’s public school system. This study examines union membership rates among Michigan’s public, charter, and private school teachers and found that while teachers in every public school district are represented by - and pay dues to - a union, only 5 out of 139 charter and 2 out of over one thousand private schools employ unionized workforces. The study concludes that school employee unions - including the Michigan Education Association and the Michigan Federation of Teachers - have powerful political and financial incentives to spend millions of dollars to prevent more parents from being able to choose non-unionized charter or private schools for their children. Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 2 OUT OF 3 Question 2: Have the instruments been demonstrated to have satisfactory reliability and validity? - 1 OUT OF 3 Question 3: Are the outcome measures clearly related to the variables with which the investigation is concerned? – 2 OUT OF 3 Question 4: Does the research design fully and unambiguously test the hypothesis? - 1 OUT OF 3 Question 5: Are the participants representative of the population to which generalizations are made? – 3 OUT OF 3 Question 6: Is the research at an advanced enough stage to make the publication of results meaningful? - 1 OUT OF 3 Question 7: Is the literature review thorough and balanced? – 1 OUT OF 3 OVERALL QUALITY SCORE: 1.57 OUT OF 3

6. The Impact of Limited School Choice on Public School Districts

Matthew Ladner, Ph.D. and Matthew J. Brouillette, August, 2000 Mackinac Center Description of Document: Case studies of how school districts in Michigan’s largest county are responding to competition from charter schools and public “schools-of-choice.” Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 3 OUT OF 3

of 41


Question 2: Have the instruments been demonstrated to have satisfactory reliability and validity? – 2 OUT OF 3 Question 3: Are the outcome measures clearly related to the variables with which the investigation is concerned? – 2 OUT OF 3 Question 4: Does the research design fully and unambiguously test the hypothesis? - 1 OUT OF 3 Question 5: Are the participants representative of the population to which generalizations are made? - 1 OUT OF 3 Question 6: Is the research at an advanced enough stage to make the publication of results meaningful? – 2 OUT OF 3 Question 7: Is the literature review thorough and balanced? – 2 OUT OF 3 OVERALL QUALITY SCORE: 1.86 OUT OF 3

7. The Cost of Remedial Education

Dr. Jay P. Greene, August, 2000 Mackinac Center Description of Document: More than a third of Michigan students leave high school without possessing basic academic skills including reading, writing, and arithmetic. This forces employers and post-secondary schools to take up the slack. This study conservatively estimates that Michigan businesses and institutions of higher education spend over $600 million annually to teach employees and students skills they should have learned in high school. The comparable national figure is $16.6 billion, but the human costs of K-12 educational failure are incalculable, according to experts’ essays included in the study’s appendices. Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 2 OUT OF 3 Question 2: Have the instruments been demonstrated to have satisfactory reliability and validity? – 2 OUT OF 3 Question 3: Are the outcome measures clearly related to the variables with which the investigation is concerned? – 3 OUT OF 3 Question 4: Does the research design fully and unambiguously test the hypothesis? – 3 OUT OF 3 Question 5: Are the participants representative of the population to which generalizations are made? – 3 OUT OF 3 Question 6: Is the research at an advanced enough stage to make the publication of results meaningful? – 3 OUT OF 3 Question 7: Is the literature review thorough and balanced? – 2 OUT OF 3 OVERALL QUALITY SCORE: 2.57 OUT OF 3

Mackinac Center Description and Evaluator’s Qualitative Ratings of the Seven Studies Classified as Interpretive Research 1. The Michigan Education Trust: A Political Economy Perspective

Dr. Peter J. Boettke, March, 1990 Mackinac Center Description of Document: Touted as cutting-edge public policy, the MET program is found to be a flawed political promise that might have to be bailed out by state taxpayers. Parents who want to save for their children’s education have better options in the private marketplace. Subsequent events led Governor Engler to scrap the program. Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 2 OUT OF 3 Question 2: Is the research at an advanced enough stage to make the publication of results meaningful? – 2 OUT OF 3

of 41


Question 3: Is the literature review thorough and balanced? – 2 OUT OF 3 OVERALL QUALITY SCORE: 2.00 OUT OF 3

2. Michigan Education Special Services Association: The MEA’s Money Machine

Andrew Bockelman and Joseph P. Overton, November, 1993 Mackinac Center Description of Document: This exhaustive report illuminates the inner workings of the Michigan Education Association’s health insurance division, known as MESSA. It documents how tens of millions of the public’s education tax dollars are wasted each year on uncompetitive teacher health insurance, and how MESSA is part of a systematic plan to subsidize the MEA’s basic operation and political activity. Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? - 1 OUT OF 3 Question 2: Is the research at an advanced enough stage to make the publication of results meaningful? – 2 OUT OF 3 Question 3: Is the literature review thorough and balanced? – 1 OUT OF 3 OVERALL QUALITY SCORE: 1.33 OUT OF 3 3. Teacher, Inc. A Private-Practice Option for Educators

Janet R. Beales, October, 1994 Mackinac Center Description of Document: This study profiles the experiences of a number of educators in private practice, and discusses the benefits that teachers, students, and schools may realize by contracting for instruction. Also included are the results from two national surveys about the legal authority of school boards to contract for instruction, and a chart to help administrators identify the fully allocated costs of in-house and contract service. Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 2 OUT OF 3 Question 2: Is the research at an advanced enough stage to make the publication of results meaningful? – 2 OUT OF 3 Question 3: Is the literature review thorough and balanced? – 1 OUT OF 3 OVERALL QUALITY SCORE: 1.67 OUT OF 3

4. Outcome-Based Education: Miracle Cure or Plague?

Dr. Bruno V. Manno, September, 1995 Mackinac Center Description of Document: This report explains why there is so much conflict over outcome-based education (OBE). When states began to institute OBE programs, they turned the crucial task of defining outcomes over to the very education establishment threatened by the process. Having adopted in principle the laudable focus on education results, the educrats then went on to propose a list of outcomes that emphasizes not academic achievement but rather, attitudes and behavior that often reflect quasi-political or ideologically correct positions.

of 41


Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 2 OUT OF 3 Question 2: Is the research at an advanced enough stage to make the publication of results meaningful? – 2 OUT OF 3 Question 3: Is the literature review thorough and balanced? – 2 OUT OF 3 OVERALL QUALITY SCORE: 2.00 OUT OF 3

5. Do Private Schools Serve Difficult-to-Educate Children?

Janet R. Beales and Thomas F. Bertonneau, October, 1997 Mackinac Center Description of Document: Private K-12 schools are sometimes criticized for accepting only those students most likely to succeed academically, and for leaving the most difficult-to-educate children to the public school system. Is this true? The diversity of private schools includes those that serve exclusively at-risk, incarcerated, or disabled children. The report describes private schools that educate each of these populations, reviews how public schools are contracting with private schools to serve difficult-to-educate students, examines policy implications including cost and school choice, and presents six case studies of Michigan private schools that serve exclusively students with special needs. Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 2 OUT OF 3 Question 2: Is the research at an advanced enough stage to make the publication of results meaningful? – 2 OUT OF 3 Question 3: Is the literature review thorough and balanced? – 3 OUT OF 3 OVERALL QUALITY SCORE: 2.33 OUT OF 3

6. The Universal Tuition Tax Credit: A Proposal to Advance Parental Choice in

Education Patrick L. Anderson, Richard McLellan, J.D., Joseph P. Overton, J.D. and Gary Wolfram, Ph.D., November, 1997

Mackinac Center Description of Document: This pathbreaking approach to expanding parental choice in education embodies a proposal to amend the Michigan constitution and establish a Universal Tuition Tax Credit (UTTC). The tax credit would offset a portion of private or public school tuition and would be claimed against state tax liabilities. In addition to improving education, the UTTC would save the state hundreds of millions of dollars per year. Unlike other tax credit plans, the UTTC would help needy families with low state tax liabilities by encouraging the creation for corporate scholarships to offset tuition costs not covered by the UTTC. The per-student credit could be claimed against the Michigan tax liability of any person or corporation. Unlike vouchers, the UTTC would not allow state funds to support religious schools, would not drain funds from the public schools, and would not spawn new entitlements or overregulation of private schools. The study includes detailed fiscal models, a discussion of school choice, a history of Michigan’s constitutional impediments to education reform, and proposed language for a constitutional amendment.

of 41


Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 3 OUT OF 3 Question 2: Is the research at an advanced enough stage to make the publication of results meaningful? – 3 OUT OF 3 Question 3: Is the literature review thorough and balanced? – 3 OUT OF 3 OVERALL QUALITY SCORE: 3.00 OUT OF 3

7. School Choice in Michigan

Matthew J. Brouillette, July, 1999 Mackinac Center Description of Document: School choice—the right, freedom, and ability of parents to choose for their children the safest and best schools—has moved front and center in the debate over how to improve education in Michigan. This three-part primer equips parents, educators, and policy makers with the facts they need to understand and advance market-based reforms that will help all Michigan schools perform at higher levels of quality and efficiency. The primer examines the history of government-funded and operated schooling, explains why nonmarket-oriented school reform efforts ultimately fail, and describes various school choice proposals including charter schools, inter-district choice, vouchers, tax credits, and universal tuition tax credits. Helpful appendices explain ways for grass-roots citizens to help advance school choice. Assessment of Social Scientific Quality of Document: Question 1: Is the research question significant, and is the work original and important? – 2 OUT OF 3 Question 2: Is the research at an advanced enough stage to make the publication of results meaningful? – 2 OUT OF 3 Question 3: Is the literature review thorough and balanced? – 3 OUT OF 3 OVERALL QUALITY SCORE: 2.33 OUT OF 3

of 41


Discussion The major questions raised in this study are discussed below in a question and answer format. Question 1: What are the major educational research concerns of the Mackinac Center? The Mackinac Center is concerned that government programs and public service unions are undermining the quality of education. As one frequent Mackinac author wrote, “The first step in understanding the state of education today is to review how government came to be the dominant force behind schooling in the United States.” 13 Indeed, much of the research of the Mackinac Center aims at revealing what it believes to be the damaging influence of government schooling and collective bargaining. Thus, in the time period under consideration, for example, out of the fourteen original and interpretive education related studies published by the Mackinac Center since 1990 seven concerned school choice. A content analysis of these studies reveals that, according to the Mackinac Center, radical deregulation is the only genuine solution to revitalizing Michigan schools. In The Universal Tuition Tax Credit: A Proposal to Advance Parental Choice in Education, Anderson et. al. writes, for example, “Choice is the engine for a market economy in all goods and services. The foundation of basic economic theory is the ability of individual consumers to choose one good over another based on their own preferences. Parents prefer good schools over poor schools for their children. Assigning children to schools, based on where students live, deprives parents of the freedom to apply their own values and priorities to selecting a school, and deprives schools of valuable marketplace incentives that drive continuous quality improvement.” 14 In the two privatization studies reviewed, the authors commissioned by the Mackinac Center argue that private schools serve difficult to educate children more efficiently and more effectively than public schools and that there is significant unused capacity in private schools. For instance, Beales and Bertonneau state “The private sector has proven an important and viable provider of education for difficult-to-educate students. In serving this diverse student population, the private sector has spawned a wide variety of schooling options to meet the needs of individual learners.” 15 This statement comes after a brief overview of private facilities in the United States that serve difficult-to-educate students and includes such private sector accommodations as home schools, shelters, halfway houses, ranches, training schools, reception centers and detention centers. Another area of interest to the Mackinac Center is the Michigan Education Association (MEA). Perhaps the title of these studies are indicative of the Center’s point-of-view: Michigan Education Special Services Association: The MEA’s Money Machine, Collective Bargaining: Bringing Education to the Table, and The Impact of School Choice on School Employee Labor Unions. Here is how Bockelman and Overton describe the Michigan Education Association:

Michigan’s most formidable union is the labor representative of some 120,000 public school employees – the Michigan Education Association…Ever since the widespread

of 41


unionization of public employees in the early 1960’s, the teachers unions have gained considerable control over public school teachers in the state of Michigan by compelling them to join their ranks. And the cost has not been cheap. School district funds are continually diverted from classrooms in order to pay for contract negotiations, strike costs, lawyer fees, litigation expenses from labor disputes, and other union prompted expenditures. 16

It is fair to say that the Mackinac Center takes a dim view of teachers unions in general and the MEA in particular. The Mackinac Center also takes a dim view of standards in Michigan’s public universities devoting two studies to the topic. To quote Bertonneau, “The state universities of Michigan, like their counterparts across the nation, are suffering from a general erosion of academic standards and a radical politicalization of the undergraduate curriculum.” 17 The problem of most concern to the Mackinac Center is that colleges and universities have become politically correct and educationally sloppy. The Center’s study Declining Standards at Michigan Public Universities concluded that self-expression and moral liberation are often emphasized over competency. The Mackinac Center also produces a series of public policy statements on a variety of educational topics ranging from outcome-based education to the unreliability of Michigan’s history textbooks. The later publication is particularly revealing of the Mackinac Center’s concerns. According to the authors of this study, “It is highly doubtful that most of Michigan’s high-school students are obtaining a basic understanding of the principles of economics from the textbooks they use in class.” 18 One of the findings of this study is that most of these texts are pro-government and fail to reveal such unknown historical facts that it was government interference that helped trigger the Great Depression and that the New Deal transferred money from tax payers to special interests. In sum, the research agenda of the Mackinac Center is consistent with its stated philosophy. They are pro-market, anti-government and deeply skeptical that teachers unions can play a role in providing better education to the children and citizens of Michigan. Question 2: Who are the Mackinac researchers and authors? The Mackinac Center has a Center staff (Appendix A), a Board of Directors (Appendix B) and a Board of Scholars (Appendix C). According to its web-site it has utilized scores of authors over the last decade. For its educational studies, the Mackinac Center tends to utilize scholars or researchers who are either employed by the Center or are staff members at similar conservative or libertarian think tanks. 19 For instance, the school choice studies include authors such as R. Wittman, who also writes for the Heartland Institute (Chicago, IL), J.R. Beales, a policy analyst for the Reason Foundation (Los Angeles, CA), J.P. Overton of the Mackinac Center, G. Wolfram of the Mackinac Center, M.J. Brouillette of the Mackinac Center and J.P. Greene of the Manhattan Institute for Policy Research (New York, NY). The authors of the privatization studies are also associated with the Mackinac Center as are the authors of the higher education studies. The public policy statements are generally written

of 41


by policy analysts from the Mackinac Center, the Reason Foundation or the Hudson Institute. A few of the Mackinac authors, such as John Chubb and Terry Moe, have national academic reputations but most of the authors do not, at least in the fields that they have chosen to write about for the Mackinac Center. Question 3: What is the process of review before publication? According to Joseph Lehman, Executive Vice President of the Mackinac Center, the Center’s process of peer-review is as follows (1) staff reviews the literature to ensure that a proposed publication will be an original contribution, (2) the publication in manuscript form is sent to a member or members of the Mackinac Board of Scholars. If there is no one on this board with sufficient expertise, the publication is sent out for review. 20 This process is not described in the publications of the Mackinac Center. Moreover, it is unclear whether or not the outside reviews are done on a blind basis. It is also not known whether or not reviewers are chosen for their substantive expertise or if other criteria are used to select reviewers. Finally, the evaluators were unable to secure a written protocol for reviewers to use when reviewing Mackinac Center publications. Question 4: What is the quality of the Mackinac Center’s original research? Seven Mackinac documents were classified as original research: (1) Michigan’s Experiment with Public School Choice: A First Year Assessment (1993); (2) Declining Standards at Michigan Public Universities (1996); (3) Collective Bargaining: Bringing Education to the Table (1998); (4) Unused Capacity in Privately Funded Michigan Schools (1999); (5) The Impact of School Choice on School Employee Labor Unions (1999); (6) The Impact of Limited School Choice on Public School Districts (2000); and (7) The Cost of Remedial Education (2000). Five of the studies were conducted between 1998 and 2000. Of the seven documents classified as original research, three were devoted to school choice, two to teacher unions, one to privatization and one to higher education. Mackinac Center original research studies employed both qualitative and quantitative methods. For instance, in the study Unused Capacity in Privately Funded Michigan Schools the researchers surveyed 342 private schools. 21 In other studies, researchers utilized telephone interviews, informal requests for information, newspapers and existing documents. The samples drawn were primarily from Michigan. Table 1 lists the average score for each index item for the seven studies taken together.

of 41


Table 1 Original Research Studies

Average Scores by Index Item and Overall Rating Question 1: Is the research question significant, and is the work original and important?

2.14

Question 2: Have the instruments been demonstrated to have satisfactory reliability and validity?

1.43

Question 3: Are the outcome measures clearly related to the variables with which the investigation is concerned?

2.29

Question 4: Does the research design fully and unambiguously test the hypothesis?

1.57

Question 5: Are the participants representative of the population to which generalizations are made?

2.17

Question 6: Is the research at an advanced enough stage to make the publication of results meaningful?

1.57

Question 7: Is the literature review thorough and balanced?

1.57

Average Score 1.82 As Table 1 indicates, the average item index score ranges from 1.43 to 2.29. The mean score for all studies is 1.82. In general, it is fair to say that the Mackinac studies tend not to be methodologically rigorous. When viewing each study individually, only two, Collective Bargaining: Bringing Education to the Table and The Cost of Remedial Education, were more than adequate. The individual scores of the studies classified as original were: 1.43 Michigan’s Experiment with Public School Choice: A First Year Assessment 1.33 Declining Standards in Michigan Public Universities 2.71 Collective Bargaining: Bringing Education to the Table 1.14 Unused Capacity in Privately Funded Michigan Schools 1.57 The Impact of School Choice on School Employee Labor Unions 1.86 The Impact of Limited School Choice on Public School Districts 2.57 The Cost of Remedial Education The range of the scores for these studies is between 1.33 and 2.71. The average of these individual study scores is 1.80. Five of seven of these studies cannot be considered methodologically rigorous according to established guidelines for social science research. Mackinac Center original research often does not create research questions that lead to testable hypotheses. For example, in their study, The Impact of Limited School Choice on

of 41


Public School Districts, Ladner & Brouillette, begin with the following quote “public education is a monopoly, and monopolies don’t work.” 22 This statement sets the tone for their research and makes it difficult for the authors to create a research question that can be tested by empirical data. According to the authors their report “seeks to ascertain whether increased competition among Michigan public schools has improved educational opportunities for children, and whether competition encourages or discourages schools to respond to the needs and demands of students and parents.” 23 They rely, however, on data collected from a single school district and support their position primarily with “anecdotal” data. For example, to support their claim that charter schools are improving other public schools in Wayne County, the authors enlist the following quote:

Despite these complaints, many superintendents express positive views about competition with charter schools. Some superintendents admitted that charter schools have forced them to pay closer attention to parents. ‘We are not ABC, CBS and NBC anymore,’ said one superintendent, implying that public schooling used to be as unassailable as the old ‘Big Three’ television networks. No longer is this the case: ‘if you see the Huns coming, you need to man the towers,’ another superintendent commented. 24

Another example can be drawn from The Unused Capacity in Privately Funded Michigan Schools. Based on a survey of 342 private schools, the Mackinac Center reported that there was additional room for over 20,000 students in the Michigan private schools responding to the survey. 25 The authors then project to the whole population of Michigan’s private schools indicating that there are approximately 55,000 additional places for students in Michigan private schools. The sample for this survey was self-selecting and self-reporting. Only one-third of the private schools in Michigan responded to the survey. It is not unreasonable to suppose that private schools hoping to increase their enrollments would respond to a survey indicating they had additional space. Moreover, there is little attempt made to determine from any other source whether these respondents are factually accurate. In the study Declining Standards at Michigan Public Universities, Bertonneau, begins as follows “The story of higher education in Michigan is beginning to resemble Hans Christian Anderson’s fairytale, The Emperors New Clothes: the naked truth is that much of the public’s money is being spent unwisely. Taxpayers, fee-payers, parents, and students are not getting the best return on their investment.” 26 The data to support this assertion were collected by the author anecdotally and by an examination of the syllabi of “key” courses. These courses include basic skill courses, teacher education courses and introductory literature and history courses. The names of these courses were ascertained by reviewing the course catalogs from 15 branches of the Michigan state university system. The author also collected individual course section’s syllabi and supplemented the analysis by examining the textbooks that were used in these courses. In addition, the author gleaned data from campus newspapers and “communicated with faculty and students.” 27 It is not clear if the author ever attended any of the classes about which he writes. It does not appear that the author had an analytic framework on which he could base his conclusions.

of 41


An examination of the bibliographic material reveals that the authors usually do not cast a wide net in terms of counter-evidence or differing points-of-view. The literature reviews of Mackinac Center authors are usually confined to those who agree with them. This strategy creates the impression that there is a substantial body of unchallenged literature that supports the Center’s findings when, in fact, there may be an extensive literature that challenges its assumptions and results. From this analysis of the quality of Mackinac Center documents classified as original studies, it is concluded that overall, the research is less than adequate according to established social science guidelines and that overall the original research studies do not meet the standard required for publication in academic journals. Question 5: What is the quality of the Mackinac Center’s interpretive research? Seven education-related Mackinac Center documents were classified as interpretive research: (1) The Michigan Education Trust: A Political Economy Perspective (1990); (2) Michigan Education Special Services Association: The MEA's Money Machine (1993); (3) Teacher, Inc. A Private-Practice Option for Educators (1994); (4) Outcome-Based Education: Miracle Cure or Plague? (1995); (5) Do Private Schools Serve Difficult-to-Educate Children? (1997); (6) The Universal Tuition Tax Credit: A Proposal to Advance Parental Choice in Education (1997); and (7) School Choice in Michigan (1999). The first interpretive research study was published in 1990 and the most recent in 1999. Of the seven studies under examination here, three were devoted to school choice, one to teachers unions, one to higher education, one to privatization and one to outcome-based education. These studies rely on reviewing secondary source materials supplemented by occasional phone calls and anecdotal data. Table 2 examined each index item for the seven studies as a whole. Table 2

Interpretive Research Studies Average Scores by Index Item and Overall Rating

Question 1: Is the research question significant, and is the work original and important?

2.00

Question 2: Is the research at an advanced enough stage to make the publication of results meaningful?

2.14

Question 3: Is the literature review thorough and balanced?

2.14

Average Score 2.09 As Table 2 indicates, the average item index score is either 2.0 or 2.14 with a mean index score of 2.09. In general, it is fair to say that while the interpretive studies are of somewhat

of 41


higher quality than the original studies, they are just adequate when judged against established social science guidelines. The individual scores of the studies classified as interpretive were: 2.00 The Michigan Education Trust: A Political Economic Perspective 1.33 Michigan Education Special Services Association: The MEA’s Money Machine 1.67 Teacher, Inc. A Private-Practice Option for Educators 2.00 Outcome-Based Education: Miracle Cure or Plague? 2.33 Do Private Schools Serve Difficult-to-Educate Children? 3.00 The Universal Tuition Tax Credit: A Proposal to Advance Parental Choice in

Education 2.33 School Choice in Michigan Overall quality scores for interpretive studies ranges from 1.33 to 3.00. The average overall quality score of these studies is 2.09. One study unambiguously met the standard of quality for social science research required for publication in a peer-reviewed academic journal. The interpretive studies raise significant questions and often address those questions in a credible manner. They are, however, weak in thoroughly reviewing the relevant literature and there is no area of research in which the interpretive research commissioned by the Mackinac Center is consistently strong. Only one study was above average against all three guidelines, The Universal Tuition Tax Credit: A Proposal to Advance Parental Choice in Education. While one might disagree with the authors’ conclusions, the study itself was thorough, analytic, and balanced. This study, however, was the exception to the rule. The other six studies were judged to be either adequate or less than adequate. In general, the interpretive studies suffer from the same limitations as the original studies. The researchers tend to begin with a conclusion, collect data in a less than rigorous manner and report results that confirm the original conclusion. This circularity of reasoning is evident throughout the interpretive studies. For instance, in the study Do Private Schools Serve Difficult-to-Educate Children, Beales and Bertonneau, begin by claiming that “The private sector, including private schools, nonpublic schools, and homeschools, offers a wide variety of education programs for this population of students.” 28 Having made this statement, the authors then examine the available data and reinforce their conclusion, not through a systematic analysis of whether or not these private options are good for children, but through a series of case studies of private sector providers. That is, the researchers go to the individuals most likely to champion private sector solutions and use these interviews as evidence of the efficiency of the private sector. There is virtually no discussion of performance outcomes outside this method. In the studies classified as interpretive, the bibliographic materials used are drawn generally from authors who support the researchers’ position. In the Mackinac Center’s interpretive studies there is little to no examination of research evidence that might challenge the conclusions drawn.

of 41


From this analysis of the quality of Mackinac Center studies classified as interpretive, it is concluded that overall, the research is less than adequate according to established social science guidelines and that overall the interpretive research studies do not meet the standard required for publication in academic journals. Question 6: What is the overall quality of the educational research published by the Mackinac Center? Overall, the quality of the educational research published by the Mackinac Center ranges primarily from inadequate to adequate. Applying the rating system established in this evaluation, the original studies under review received a 1.80 on a scale of one to three with three being the highest. Interpretive studies received a 2.09 using the same scoring system. Few of the studies reviewed are exceptional. Very few Mackinac Center sponsored studies would be accepted for publication in a peer-reviewed academic journal. The studies evaluated in this research typically had five elements in common: (1) a preconceived position is taken, (2) an unclear or untestable hypothesis is created, (3) data are collected in a less than rigorous manner, (4) these data are presented in an anecdotal style, and (5) the results are reported with certainty. Clearly, the Mackinac Center has a belief system and a political agenda that shapes its research. It is worth noting that of the twenty-two studies the Mackinac Center lists, only fourteen can reasonably be considered studies in the social scientific meaning of the term. It is evident that the Mackinac Center belief system strongly influences its research agenda and the research methodologies employed by the authors of its studies. Thus, Center reports tend to use social science language without proper social science methods in a way that gives the appearance of social scientific legitimacy to the Center’s preconceived beliefs and ideas. This being the case, documents described by the Mackinac Center as studies should be regarded with considerable caution by the public, policy makers and political leaders.

of 41


Implications Any institution offering solutions to the manifold challenges facing American education should be held to a high standard. Has the research of the Mackinac Center enabled us to think more clearly and make better policy decisions about education? Put differently, if the Mackinac Center’s research did not exist would we know more or less about the educational options available to the citizens of Michigan? The analyses in this report suggest that much of the work of the Mackinac Center retards rather than advances public understanding of the issues that it has addressed. In the area of school choice, for instance, the work of the Mackinac Center systematically ignores evidence that charter schools may reinforce the social stratification of schools; no where in Mackinac sponsored research is there any evidence that school choice may lead to re-segregation; no where in Mackinac sponsored research is there any indication that the record of school choice plans in promoting student achievement is mixed and ambiguous; and no where in Mackinac sponsored research is there any indication that there is evidence that suggests privatizing reforms have had little effect on student achievement. The way in which Mackinac Center sponsored research characteristically frames questions is biased and the methodology employed of little social science merit. It is, therefore, important that the education related research of the Mackinac Center be carefully and independently evaluated before any serious discussion of the proposals put forward takes place. Over the past two decades, reports produced by national and state private think tanks and policy organizations have played an increasingly important role in shaping education policy. These reports are issued often without being subjected to the kind of peer-review and expert scrutiny that is an important benchmark of social science research and that is a requirement for publication in journals such as Educational Evaluation and Policy Analysis or American Educational Research Journal. While it is possible to conduct high quality social science research in private think tanks and research centers, it is necessary that the studies are subjected to an internal review process that has integrity and that they be scrutinized by qualified and disinterested external reviewers. The evaluation of Mackinac Center sponsored education research presented here indicates that the Center often promotes research findings to policy makers and the public before the research can be reviewed by qualified, disinterested external reviewers. The widespread influence of policy reports issued outside of the peer-review process therefore, threatens to subvert sound policy by promoting priorities that may not rest on solid social science research or draw on the best available research knowledge. Often policy reports issued by private think tanks are prominently featured in media stories, whereas any scholarly response normally occurs much later and is commonly little noticed by either the press or by policy makers. By flooding the media with “studies,” organizations such as the Mackinac Center attempt to influence the public agenda. When the media pick up these reports and pass the “evidence” onto readers or viewers uncritically, it undermines the public’s ability to understand the issues confronting public education. As evidence of this, we might cite the seeming anomaly that parents consistently report being satisfied with the performance of their local public

of 41


school but believe that public education in general is much less successful. 29 What parents are able to see for themselves they often consider laudable; what they hear about other schools from such sources as the Mackinac Center, is often relentlessly disparaging. Scholars are partially responsible for the unchallenged dissemination of advocacy reports cloaked in the language of social science research. The weakness of a particular advocacy piece may be pointed out in a scholarly journal months after the original well-publicized report is released. The audience for these scholarly journals is, however, generally narrow and specialized and these publications tend to have little direct impact on the school reform debate. This state of affairs suggests that the public would be better served if scholars became more interested in the publications of institutions such as the Mackinac Center and communicated in a way that is easily understood by the lay reader. It is clear that both public policy makers and the media would benefit by an educational process that allowed them to be more discriminating consumers of research.

of 41


Recommendations In the marketplace of ideas it is important that research standards be made public and carefully maintained so that ideology cannot so easily disguise itself as social science. Therefore, we recommend that the Michigan legislature provide funds to:

1. Establish a network of education experts with diverse expertise who are willing, in a timely fashion, to write reviews of privately produced documents and to allow those reviews to be distributed to policy makers and the public;

2. Create and maintain a system for monitoring the work produced by private think tanks to facilitate the timely review of reports they issue; and

3. Support a web-site containing a database indicating the social science value of the hundreds of studies being published by think tanks and policy centers. This web-site would assist legislators and the public in judging the quality of the research that is being used to urge them to take a particular course of action.

For the citizens of Michigan it is important that the work of the Mackinac Center be seen for what it is. The Center is devoted to privatizing state institutions and to deregulating public education. The evidence presented in Mackinac Center studies is often weak and at times misleading. It is hoped that this report has been helpful in revealing the shortcomings and the possible dangers inherent on basing public policy on the research of the Mackinac Center.

of 41


Appendix A

Mackinac Center Policy Staff Aussicker, David Vice President for Advancement Bardallis, David Managing Editor of Publications Brouillette, Matthew Director of Education Policy Coulson, Andrew Senior Fellow in Education Policy Davis, Kent Science Advisor Gifford, Mary Director of Leadership Development Hunter, Robert Director of Labor Policy LaFaive, Michael Research Project Manager Lehman, Joseph Executive Vice President Moser, Elizabeth Education Research Assistant Overton, Joseph Senior Vice President Reed, Lawrence President Shrode, Kendra Assistant to the President Walker, Samuel Communications Specialist

of 41


Appendix B

Mackinac Center Board of Directors Antonini, Richard - Retired President of Foremost Insurance Company Cook, Peter - Chairman of the Board, Cook Holdings, LLC Gadola, Paul - U.S. District Judge Haworth, Richard - Chairman of the Board, Haworth Incorporated Letica, Mara - Executive Vice President and General Counsel, Letica Group Levy, Edward - President, C. Levy Comany Lockwood, Rodney - President, The Lockwood Group Maguire, Bruce - Chairman, Wolverine Development Corporation Maguire, Joseph - President, Wolverine Development Corporation McLellan, Richard - Member, Dykema Gassett Olson, D. - Vice President and General Counsel, Amerisure Companies Rodney, James - President of the Rodney Fund Shinkle, Linda - Attorney at Law

of 41


Appendix C

Mackinac Center Board of Scholars Alexander, Donald Attarian, John Bertonneau, Thomas Birzer, Brad Boettke, Peter Bornhofen, John Browne, William Colarelli, Stephen Crocker, Keith Crowner, Robert Cutler, Richard Daddow, Robert Dresch, Stephen Ebeling, Richard Edgens, Jefferson Esposto, Alfredo Gardner, Wayland Haywood, Dale Heberling, Michael Hook, Ormand Hutchison, Harry Janda, David Kirk, Annette

Kleiman, Robert Littmann, David Matcheck, Dale McCracken, Paul Nastas, George Pafford, John Perry, Mark Potter-Witter, Karen Rehmke, Gregory Safranek, Stephen Schimmel, Louis Sheehan, James Sirico, Robert Skoppek, Jurgen Smith, Bradley Taylor, John Vedder, Richard Veryser, Harry Walter, John Wilson, William Wing, Martin Wolfram, Gary

* Full bios for each member of the Mackinac Center’s Board of Scholars may be found on the Center’s web-site at http://www.mackinac.org/people.asp?TypeID=1

of 41


Appendix D

Mackinac Center Descriptive Overview of Opinion Essays 1. Educational Choice for Michigan

Lawrence Reed and Harry Hutchison (Editors and Co-authors), September, 1991 A collection of five articles: A Call for Educational Renewal by Paul DeWeese, Educational Choice by John E. Chubb and Terry M. Moe, Private Schools: Let Competition Heat Up by Harry Hutchison, A Focus on Detroit by Lawrence Reed and Harry Hutchison and The Time for Real Choice Has Arrived by Lawrence Reed. The sad state of public education in Michigan and America is largely due to its organization as a government-protected monopoly. The authors argue that injecting choice, competition and accountability into education would result in dramatic improvement. The report explodes the myths that the problem in education is too little money and that choice would lead to segregation or elitism. One chapter focuses on the remarkable achievements of 107 non-public schools in Detroit. 2. The Need For Debt Policy in Michigan Public Schools

Michael Arens, March, 1998 Public school construction is booming across Michigan, but due to citizens’ negative perceptions, many districts are finding it harder and harder to gain voter approval for bond proposals to find needed projects. This analysis of Michigan public school bonding concludes that development of formal debt policies can help schools earn essential voter trust by managing bond monies in the most efficient and effective manner. The report recommends fifteen elements for a sound debt policy that school districts should adopt to avoid common pitfalls and problems in bonding, including excessive borrowing, improper accounting, and conflict of interest in debt issuance. 3. Keeping Michigan on Track: A Blueprint for Governor Engler and the 90th

Legislature N/A, January, 1999

The close of the twentieth century finds Michigan in a position that seemed impossible barely a decade ago: record low unemployment, a thriving economy, growing educational opportunities, and a sense of accomplishment and high spirits. But much can be done to make Michigan an even better place to live and work. This report’s five sections offer the Governor and the Legislature 41 specific recommendations that will strengthen property rights protection, reform labor law to protect worker rights, improve education for Michigan children, spur law to protect worker rights, improve education for Michigan children, spur economic growth and development, and enhance the state’s transportation infrastructure. 4. How Reliable Are Michigan High School Economics Textbooks?

Burton Folsom, George Leef, Dirk Mateer, June, 1999

of 41


A strong knowledge of sound economic principles is not only important in the twenty-first century global marketplace, it is essential for the maintenance of a free society. Are Michigan high school students being taught what they need to know in order to succeed and prosper? This review of 16 of the most commonly used economics textbooks in Michigan high schools uses 12 criteria - including issues of trade, taxation, and the role of government - to evaluate which texts are and are not effective at presenting students with a balanced and accurate perspective on the modern market economy. Each text is graded, from A to F, on its ability to clearly instruct students in the “economic way of thinking.”

of 41


Appendix E

Mackinac Center Descriptive Overview of Administrative and Legislative Guides

1. Modern Schools for Michigan N/A, September, 1993

After the Michigan Legislature’s vote to end property tax funding for schools, the Mackinac Center issued this outline of a comprehensive education overhaul for Michigan. It suggests policy changes including parental choice, school-based management, and charter schools. Also included is an explanation of the Mackinac Center’s innovative Education Credit Account idea. 2. Making Schools Work: Contracting Options For Better Management

Janet R. Beales and John O’Leary, January, 1994 Can America’s public schools be improved? Unquestionably. Without additional spending, school administrators can take advantage of the expertise of the private sector, introducing innovations that will make a world of difference. This study reveals dozens of examples of private companies now providing management, instructional, and support services to public schools across America. A must-read for anyone interested in changing public education by putting competition and the profit motive to work. 3. Doing More With Less: Competitive Contracting For School Support Services

Janet R. Beales, November, 1994 Competitive contracting can provide schools with expertise, flexibility, and cost efficiencies not always available with in-house service provision. If they are properly designed and monitored, contracts between schools and private providers can help school administrators do more with less. Includes step-by-step guidelines for the “make or buy” decision, tells how to avoid pitfalls, and suggests measures for contractor evaluation. 4. Are Michigan’s History Textbooks Reliable?

Dr. Burton W. Folsom, January, 1996 When history texts are poorly written, students are merely bored. But when they are distorted and biased, students may act on false ideas and live out a lie. How reliable are Michigan’s history texts at presenting the past in ways which are well organized, accurate, clear, and free of bias? In this study, four Michigan history textbooks are analyzed and reviewed against these criteria. The reliability of these texts is especially important since Michigan history is a required subject for Michigan fourth graders, and it is studied by many junior high and high schoolers too. Parents, teachers, and school officials will find this study a valuable tool for making the best choices for their students.

of 41


Notes 1. Mackinac Center for Public Policy [world wide web page]. Available from

http://www.mackinac.org.bio.asp? 2. Mackinac Center for Public Policy [world wide web page]. Available from

http://www.mackinac.org/bio.asp?ID=3 3. Mackinac Center for Public Policy [world wide web page]. Available from

http://www.mackinac.org/1662 4. Mackinac Center for Public Policy [world wide web page]. Available from




http://www.mackinac.org 8. Mackinac Center for Public Policy [world wide web page]. Available from

http://www.mackinac.org/article.asp?ID=3469 9. Molnar, A., April 11, 2001. The media and educational research: What we know vs.

what the public hears. Presented at the 2001 AERA annual meeting, Seattle, WA. Available from www.uwm.edu/Dept/CERAI/edpolicyproject/cerai-01-14.html.

10. Ibid. Recent articles and essays addressing the issue include: “Oops, Sorry: Seems That My Pie Chart Is Half-Baked,” Patricia Cohen, New York Times, April 8, 2000; “Reckonings: How to Be a Hack,” Paul Krugman, New York Times, April 23, 2000; “They Blinded Me with Political Science: On the Use of Non Peer-Reviewed Research in Education Policy,” Ed Muir, PS, December, 1999; “Without Peer,” Christopher Shea, Lingua Franca, April 2000; “Should Public Discussion of PS Research Be Controlled? Why Interest Group Recommendations on the Proper Procedures for Reporting Research Should be Treated With Skepticism,” Paul Peterson, PS, June, 2000.

11. The complete list was compiled from the Mackinac Center for Public Policy web-site. 12. American Psychological Association (1995). Publication Manual of the American

Psychological Association. Washington, DC: American Psychological Association. 13. Brouillette, Matthew. “School Choice in Michigan: A Primer for Freedom in

Education. ” Mackinac Center for Public Policy, July 1999. 14. Anderson, Patrick, et al. “The Universal Tuition Tax Credit : A Proposal to Advance

Parental Choice In Education.” Mackinac Center for Public Policy, November, 1997.

15. Beales, Janet and Thomas Bertonneau. “Do Private Schools Serve Difficult-to-Educate Children.” Mackinac Center for Public Policy, October, 1997.

16. Bockelman, Andrew and Joseph Overton. “Michigan Education Special Services Association: The MEA’s Money Machine.” Mackinac Center for Public Policy, November, 1993.

17. Bertonneau, Thomas. “Declining Standards at Michigan Public Universities.” Mackinac Center for Public Policy, October, 1996.

of 41


18. Folsom, Burton and George Leef and Dirk Mateer. “How Reliable are Michigan High School Economics Textbooks?” Mackinac Center for Public Policy, June, 1999.

19. National Committee for Responsive Philanthropy, March 12, 1999. $1 billion for ideas: Conservative think tanks in the 1990’s, www.ncrp.org/psr/pressreleases/thinktanks.htm and July 1997, Moving a public policy agenda: The strategic philanthropy of conservative foundations, www.ncrp.org/reports/moving.htm.

20. Phone interview with Joseph Lehman, Executive Vice President, The Mackinac Center, August 30, 2001 and follow-up e-mail on September 13, 2001. As of September 15, 2001, no response has been received from Mr. Lehman. Questions include: (1) Is your peer-review process "blind" - ie. is the name of the author revealed to the reviewer? (2) How many reviewers review a manuscript? (3) What criteria are used to select the Board reviewers? (4) What criteria are used to select the outside reviewers if there is no one on the Board with sufficient expertise? (5)Is there a written protocol for reviewers to use when reviewing Mackinac Center publications? If yes, might it be possible to have a copy? (6) Finally, have manuscripts been rejected as a result of the peer-review process and if yes, what percentage of the total number of manuscripts does this represent?

21. Brouillette, Matthew. “Unused Capacity in Privately Funded Michigan Schools.” Mackinac Center for Public Policy, March, 1999.

22. Ladner, Matthew and Matthew Brouillette. “The Impact of Limited School Choice on Public School Districts.” Mackinac Center for Public Policy, August, 2000.

23. Ibid. 24. Ibid. 25. Brouillette, Matthew. “Unused Capacity in Privately Funded Michigan Schools.”

Mackinac Center for Public Policy, March, 1999. 26. Bertonneau, Thomas. “Declining Standards at Michigan Public Universities.”

Mackinac Center for Public Policy, October, 1996. 27. Ibid. 28. Beales, Janet and Thomas Bertonneau. “Do Private Schools Serve Difficult-to-

Educate Children.” Mackinac Center for Public Policy, October, 1997. 29. Gallup, A. M. and Rose, L. C., downloaded September 4, 2001. The 33rd annual Phi

Delta Kappa/gallup poll of the public’s attitudes toward the public schools. Phi Delta Kappan, http://www.pdkintl.org/kappan/k0109gal.htp.

of 41


About the Authors Peter W. Cookson, Jr. teaches at Teachers College, Columbia University and has published widely in the area of school choice and school improvement. He has authored and/or co-authored numerous books and articles on educational policy and reform. He is currently completing a book about charter schools. Alex Molnar is Professor and Director of the Education Policy Studies Laboratory, Department of Educational Leadership and Policy Studies, College of Education, Arizona State University. He has published widely on education reform topics and is a principal investigator for the Wisconsin Student Achievement Guarantee in Education (SAGE) class size reduction program. Katie Embree holds an Ed.D. in Higher Education Administration from Teachers College and an MBA in Finance from Seton Hall University. She has worked on a number of education evaluations.

let the buyer beware - great lakes centertimely fashion, to write reviews of privately produced...

Documents