Social programming is an evolved mechanism for addressing social needs (among other things). Evaluation focuses largely on social programs. We review selected aspects of the complex dynamic, political, and contextual factors that influence social programming, andfrom these we draw conclusions about the conduct of emergent realist evaluation.

Social Programming and Policy- Making: A Realist Perspective Melvin M . Murk, Gary T. Henry

The field of evaluation today largely remains focused on program evaluation. Program evaluation, of course, is fundamentally concerned with social policies and programs. Nevertheless, many early writers about program evaluation gave relatively little attention to the nature of social policy-making or programming (Shadish, Cook, and Leviton, 199 1). This inattention was unfortunate, because implicit or explicit beliefs about the nature of social programming and policy- making influence the ways evaluators define their purpose, approach their tasks, design and carry out evaluations, communicate their findings, and seek to facilitate use (Shadish, Cook, and Leviton, 1991; also see Henry and Rog on evaluation use in Chapter Five of this volume).

In this chapter, we discuss social programming from the perspective of emergent realist (ER) evaluation, and we sketch several implications for eval- uation practice. The discussion will apply the twin concepts of sensemaking and valuing (which were presented in Chapter Two concerning knowledge construction (Julnes and Mark) and in Chapter Three on valuing (Henry and Julnes)), to the analysis of social programming and policy-making. By consid- ering the nature of social programming and policy-making, this chapter also provides a contextual argument for using the realist approach sketched out in Chapter One of this volume.

We address both policy-making and social programming, though with more explicit attention to the latter. We do not address other evaluands, such as personnel and products, that might be addressed by the transdiscipline of evaluation that Scriven (1993) describes.

Anderson (1990, pp. 5-8) casts the notion of policy-making widely, including (1) the needs or policy demands that are to be addressed, (2) the

NEW DIRECTIONS FOR EVALUATION, no. 78, Summer 1998 0 Jossey-Bass Publishers 73

86 REALIST EVALUATION: AN EMERGING THEORY IN SUPPORT OF PRACTICE

what conditions, and why? A final implication we will draw from our selective review of social programming is the appropriateness, in ER evaluation, of empirically studylng and critically examining values. The very process of defin- ing a condition as a social problem worthy of collective action involves valu- ing as well as sensemaking. Similarly, the processes through which social programs are created and revised involve valuing as well as sensemaking. In addition, the values of the social programming world influence evaluation, for example, in the selection of outcome measures, and the values of evaluations can likewise influence the social programming world.

In short, a review of several key characteristics of social programming and policy-making supports many of the attributes of emergent realist evaluation presented throughout this volume, including its emphasis on the dual tasks of sensemaking and valuing, the importance of seeking enlightenment use through the study of underlylng generative mechanisms, the value of the molar versus molecular distinction, and the utility of studying variations in imple- mentation and assessing what works for whom in what circumstances.

,

References

Anderson, J. E. Public Policymahing. Boston: Houghton Mifflin, 1990. Bhaskar, R. A. A Realist Theory of Science. Atlantic Highlands, N.J.: Humanities Press,

Bickman, L. (ed.). Advances in Program n e o r y . New ctions for Program Evaluation, no.

Chen, H.-t. Theory-Dviven Evuluations. Newbury Park, Calif.: Sage, 1990. Cohen, D. K. “Standards-Based School Reform: Policy, Practice, and Performance.” Paper

presented at the Brooks Conference on Performance-Based Approaches to School Reform, Washington, D.C., 1995.

Cook, T. D. “Dilemmas in the Evaluation of Social Programs.” In M. B. Brewer and B. E. Collins (eds.), Scientijic Inquiry and the Social Sciences: A Volume in Honor of Donald 7. Campbell. San Francisco: Jossey-Bass, 198 1.

Corcoran, D., and Goertz, M. “Institutional Capacity and lhgh-Performance Schools.” Edu- cational Researcher, 1995,24 (9), 27-31.

Cronbach, L. J. Designing Evaluations of Educational an8 Social Programs. San Francisco: Jossey-Bass, 1982.

Cronbach, L. J., and others. Toward Reform of Program Evaluation. San Francisco: Jossey- Bass, 1980.

Davis, M., Hall, T., and Henry, G. Report on the bpnditure ofhttery Funds, Fiscal Year 1995. Atlanta, Ga.: Council for School Performance, 1995.

Einhom, H. J., and Hogarth, R. M. “Judging Probable Cause.” Psychological Bulletin, 1986,

Etzioni, A. “Incorrigible.” The Atlantic Monthly, July 1, 1994, 74, pp. 14, 16. Fountain, J. E. “Comment: Disciplining Public Management Research.” Journal of Policy

Glazer, N, “How Social Problems are Born.” f i e Public Interest, 1994, 115, 31-44. Gueron, J. M. “Learning About Welfare Reform: Lessons from State-Based Evaluations.” In

D. J. Rog and D. Fournier (eds.), Progress and Future Directims in Evaluations: Perspectives on Theory, Practice, and Methods. New Directions for Evalualion, no. 76. San Francisco: Jossey-Bass, 1997.

1978.

47. San Francisco: Jossey-Bass, 1990.

99, 3-19.

Analysis and Management, 1994, 13 (2), 269-277.

’ -

SOCIAL PROGRAMMING AND POLICY-MAKING 87

Henry, G., and Bugler, D. Evaluation ofthe HOPE Scholarship Program. Atlanta, Ga.: Council for School Performance, 1996.

Hossler, D., Braxton, J., and Coppersmith, G. “Understanding Student College Choice.” In J. Smart (ed.) Higher Education: Handbook of Theory and Research. New York: Agathon Press, 1989.

House, E. R. “Realism in Research.” Educational Researcher, 1991,20, 2-9. Kahneman, D., Slovic, P., and Tversky, A. (eds.). Judgment Under Uncertainty: Heuristics and

Biases. New York: Cambridge University Press, 1982. Kinder, D. R., and Sears, D. 0. “Prejudice and Politics: Symbolic Racism Versus Racial

Threats to the Good Life.”Joumal of Personality and Social Psychology, 1981,40,414-431. Kingdon, J. Agendas, Alternatives, and Public Policies. (2nd ed.) Boston: HarperCollins, 1995. Koenig, 1. W. f i e Chief Executive. New York: Harcourt, Brace, 1975. Lynn, L. E., Jr. “Public Management Research: The Triumph of Art over Science.”Journal of

Mackie, J. L. “Causes and Conditions.” American Philosophical Quarterly, 1965, 2 (4),

Mackie, J. L. The Cement of the Universe: A Study of Causation. Oxford: Clarendon, 1974. March, J. G,. and Simon, H. A. Organizations. New York: Wiley, 1958. Marcon, R. “Doing the Right Thing for Children: Linking Research and Policy Reform in

the District of Columbia Public Schools.” Young Children, 1994,2-10. Mark, M. M. “Validity Typologies and the Logic and Practice of Quasi-Experimentation.”

In W.M.K. Trochim (ed.), Advances in Quasi-Experimental Design and Analysis. New Direc- tions for Program Evaluation, no. 31. San Francisco: Jossey-Bass, 1994.

Policy Analysis and Management, 1994,13 (2), 231-259.

245-2 55,26 I -2 64.

Murray, C. Losing Ground: American Social Policy 1950-1980. New York: Basic Books, 1984. Nelson, B. J. Making an Issue of ChiZd Abuse. Chicago: University of Chicago Press, 1984. Patton, M. Q. “The Evaluator’s Responsibility for Utilization.” Evaluation Practice, 1988, 9,

Patton, M. Q. Utilization-Focused Evaluation. (3rd ed.) Thousand Oaks, Calif.: Sage, 1997. Pawson, R., and Tilly, N. Realistic Evaluation. Thousand Oaks, Calif.: Sage, 1997. Popper, K. The Poverty of Historicism. London: Routledge, 1957. Popper, K. In Search uf a Better World. New York: Routledge, 1994. Pressman, J. L., and Wildavsky, A. Implementation. (3rd ed.) Berkeley: University of Cali-

fornia Press, 1984. Rosenau, J. Citizenship Between Elections: An. Inquiry into the Mohilieable American. Macmil-

lan, 1974. Scheirer, M. A. “Program Theory and Implementation Theory: Implications for Evaluators.”

In L. Bickman (ed.), Using Program Theory in Evaluation. New Directions for Program Evaluation, no. 33. San Francisco: Jossey-Bass, 1987.

Scher, S. “Congressional Committee Members as Independent Agency Overseers: A Case Study.” American Political Science Review, 1960,54, 91 1-20.

Scholz, J. T., Twombly, J., and Headrick, B. “Street-Level Political Controls over Federal Bureaucracy.” American Political Science Review, 1991,85 (3), 829-850.

Scriven, M. S. Hard-Won Lessons in Program Evaluation. New Directions for Program Eval- uation, no. 58. San Francisco: Jossey-Bass, 1993.

Shadish, W. R., Cook, T. D., and Leviton, L. C. Foundations of Program Evaluation: Theories of Practice. Newbury Park, Calif.: Sage, 1991.

Suchman, E, A. Evaluative Research: Principles and Practice in Public Service and Social Action Programs. New York: Russell Sage Foundation, 1967.

Weiss, C. H. “Do We Know Any More About Evaluation Use?” Plenary address at annual meeting of the American Evaluation Association, San Diego, Calif., 1997.

Weiss, C. H. “If Program Decisions Hinged Only on Information: A Response to Patton.” Evahation Practice, 1988b, 9, 15-28.

Weiss, C. H. “Evaluation for Decisions: Is Anybody There? Does Anybody Care?” Evalua- tion Practice, 1988a, 9, 5-20.

5-24.

88 REALIST EVALUATION: AN EMERGING THEORY IN SUPPORT OF PRACTICE

Weiss, C. H. “Research for Policy’s Sake: The Enlightenment Function of Social Research.”

Weiss, C. H., and Bucuvalas, M. J. Social Science Research and Decision-Making. New York:

White, L. “Policy Analysis as Discourse.” Journal of Policy Analysis and Management, 1994,

Wholey, J. S. Evaluation and Effective Public Management. Boston: Little, Brown, 1983. Witkin, B. R., and Altschuld, J. W. Planning and Conducting Needs Assessments: A Practical

Wood, B. D., and Waterman, R. W. “The Dynamics of Political Control of the Bureaucracy.”

Wright, J. D. Address Unknown: The Homeless in America. New York: A. de Gruyter, 1989.

Policy Analysis, 1977,3, 531-545.

Columbia University Press, 1980.

13 (3), 506-525.

Guide. Thousand Oaks, Calif.: Sage, 1995.

American Political Science Review, 1991, 85 (31, 801-828.