recruiting the next generation of measurement professionals

Recruiting the N e x t Generation of Measurement Professionals Stephen G . Sireci, University of Massachusetts at Amherst

How are we going to find the next generation of measurement professionals? What are the issues pertinent to recruitment and training?

If there were any period one would desire to be born in, is it not the age of revolution; when the old and the new stand side by side and admit of being compared; when the energies of all . . . are searched by fear and by hope; when the historic glories of the old can be compensated by the rich possibilities of the new era? This time, like all times, is a very good one, if we but know what to do with it. (Ralph Waldo Emerson, Phi Beta Kappa Address, 1837.)

t the 1999 annual conference of A the National Council on Mea- surement in Education (NCME), I was asked to share my thoughts on the future of graduate training in educational measurement. This article is an evolution of the talk I prepared for that day. Since that time, two developments have refined my thoughts on this issue. First, I became chair of the NCME Recruitment of Educational Measurement Specialists Committee. Second, I received illuminating feed- back from several reviewers on my ini- tial paper. Like many of our colleagues (Brennan & Plake, 1991; Patelis, Kolen, & Parshall, 1997), I believe the shortage of measurement professionals is extremely troubling. I also believe the lack of people of color in our profession is disheartening, and inhib- its the growth of our profession. How- ever, as a professor of educational measurement, I see many positive developments that could greatly change the size and diversity of our professional community. In this article, I briefly discuss the present shortage of measurement professionals and make some suggestions for actions the measurement community could take to increase the number and diversity of

educational measurement professionals.

Educational Measurement Past and Present Modern educational measurement can trace its origins to the field of psychometrics. Webster’s Ninth New Colle- giate Dictionaq (Merriam-Webster, 1988) defines psychometrics as the “psychological theory or technique of mental measurement” (p. 951). Some researchers mark the beginning of psychometrics with the psychophysical experiments conducted by Weber and Fechner during the latter half of the 19th century. Others trace the origins back much further to the “civil service’’ testing programs used in China over 4000 years ago (Kaplan & Sac- cuzzo, 1982). Although there is some debate about the genesis of psychometrics (Englehard, 1997), there can be no doubt that psychometrics became a legitimate science, and educa- t ional measurement became a legitimate profession, during the 20th century. The beginning of the 20th century witnessed the first large-scale intelligence tests (the Binet-Simon scales [1905] and their English adap- tations), the birth of a “test scoring machine” (Reynold Johnson’s IBM 805 scanner, 1936), and two formal para- digms for modeling mental measure- ments: classical test theory and item response theory. The ubiquitous mul- tiple-choice item is also a 20th century invention. Consider for example: Which of the following measurement specialists made their most significant professional contributions during the twentieth century?

(a) Cronbach (b) Likert

(c) Lindquist (d) Lord (e) Samejima (f) all of the above. The obvious correct answer illus-

trates that the 20th century was a time of rapid growth for the science and practice of mental measurement. As we begin the 21st century, clearly, there is no slowing of progress in our field. New measurement models, new assessment formats, and new opportunities are emerging. Those who doubt this rapid and constant progress need only review the most recent NCME conference programs. Examples of ad- vances in test theory and application are everywhere.

Rapid progress can be stressful on a profession, Contemporary measurement specialists must keep up with a growing number of technological developments, and with increasing de- mands from consumers of educational tests. Innovations in computer-based testing, the need to test individuals who operate in different languages, and the need to test traits previously thought to be untestable (e.g., inter- personal skills) are a few examples of challenges facing today’s psychometri- cian. In addition, there is an enormous demand for professionals with measurement expertise.

The beginning of the 21st century could be described as the days ofac- countability . The federal government wants to make sure educational programs funded with federal monies (e.g., Chapter I, Challenge Grants, etc.) have a positive impact on student

Stephen G. Sireci, Ph.D. is Associate Professor, Department of Educational Policy, Research, and Administration, School of Education, University of Massa- chusetts, Amherst, MA 01003-4140. His specialkations are test development, test evaluation, cross-lingal assessment, and multidimensional scaling. E-mail can be sent to [email protected].

Winter 2000 5

achievement. States want to make sure their schoolchildren meet ex- pected standards of performance. Na- tional policy makers want to make sure our schools teach students the skills they need to compete in a global society. Corporations want to make sure they hire and promote the appro- priate employees. States want to make sure their schoolteachers possess the knowledge, skills, and abilities necessary to be a “good teacher.” Colleges, universities, and post-graduate institutions want to select the best students. Such “assurances” require valid out- come measures. Such assurances require measurement specialists. One need only to pick up the paper, turn on the radio, or watch television to find evidence that educational assessment is fully integrated with our capitalistic, industrialized, and market-driven economy. Measurement specialists are in great demand and are working in increasingly diverse settings.

The Shortage of Measurement Professionals The economical law of supply and demand is simple: when demand for something is high, supply is usually in- creased. However, many measurement professionals have pointed out that while the demand for measurement expertise is high, the supply continues to be low (Brennan & Plake, 1991; Pat- elis, Kolen, & Parshall, 1997). The shortage of measurement specialists is worrisome for two reasons. First, if the supply continues to be low, the future of the profession and progress in the science of educational measurement could be jeopardized. Second, there is too much work out there for the present number of psychometricians. Lots of tests, coupled with few psychometricians, could lead to poor measurement practices.

Thus, the current state of affairs is high demand for measurement expertise, with a low supply of measurement experts. Will this trend continue for the foreseeable future? Although it is difficult to predict the future, it is important for us to realize that we have some control over the destiny of our profession. The collective actions we take today will impact our tomorrow. The Emerson quote at the beginning of this article speaks of a new era with rich possibilities. I believe the quote describes our current

situation in educational measurement. I believe we have the power to increase the number and diversity of educational measurement professionals-if we put our minds and efforts to the task. In the next section, I suggest five activities in which we could engage to help accomplish the goals of increasing the number and diversity of educational measurement professionals. These suggestions are based on some current trends and developments I see as a professor and practi- tioner in the field of educational measurement.

Suggestions for Increasing the Number and Diversity of Educational Measurement Specialists I believe there are five major activities in which we should engage to recruit new and diverse professionals into our field. Specifically, we need to:

1. Familiarize people working and studying in mathematics and the social sciences with the field of educational measurement;

2. Reach out to people of color and make the profession more open and attractive to them;

3. Engage testing organizations, national, state, and local educational associations (governmental and not-for-profit), credentialing organizations, and for-profit companies in the funding and supporting of graduate students in educational measurement;

4. Initiate, advertise, and support nondoctoral degrees in educational measurement; and

5 . Develop distance-based training programs in educational measurement.

Each of these activities will be dis- cussed in turn.

Familiarize People in Mathematics and the Social Sciences with Educational Measurement We work in a very interesting field. I believe our best recruiting tool is ex- posing others to this field. Very few people know our profession exists, which obviously precludes them from considering educational measurement as a career option. To introduce potential colleagues to our field, and to accomplish the goal of a more diverse membership, we must reach out to undergraduate students in mathematics

and the social sciences, K-12 teachers and administrators, educational policy professionals, and people of color throughout the educational research community.

The most serious problem for us to overcome in recruiting new professionals is that most undergraduates have never heard of psychometrics or educational measurement. This oblivi- ousness is ironic, because virtually all of them have spent hours studying for and taking the tests we develop. The NCME Recruitment Committee is tackling this problem, but help is needed. Our recent activities include the development of an informative brochure about careers in measurement for undergraduates, a compila- tion of a list of graduate programs in educational measurement, and a com- pilation of a list of internships, fellow- ships, and assistantships associated with graduate study in this area. These documents are available on the NCME web site (http:Nncme.ed.uiuc.edu). Please visit this site and share the information with anyone who needs to hear about us. In addition, the Com- mittee is developing an ongoing com- pilation of stories by measurement professionals about how they entered the profession. If you would like to contribute to this volume, please send your story to the address listed in the note at the end of this articleq2 Finally, the Committee is considering the development of a video or CD-ROM to also share information about our field with the masses. These ideas are exciting, but we realize that more action is necessary to spread the word about our interesting and exciting profession.

With respect to reaching out to K-12 teachers, it has long been known that they spend a large portion of their time dealing with educational assessments, but that they receive very little training in this area (American Fed- eration of Teachers [AFT], NCME, & National Education Association (NEA), 1990; Impara & Plake, 1995; Nolen, Haladyna, & Haas, 1992; O’Sullivan & Chalnick, 1991; Popham, 1991, Schafer, 1991). Therefore, intro- ducing these professionals to our field is straightforward: make instruction in educational measurement a mandatory part of the teacher education curriculum. The Standards for Teacher Competence in the Educational As- sessment of Students, developed

6 Educational Measurement: Issues and Practice

jointly by the AFT, NCME, and NEA, was a step in the right direction, but this document has not had the impact intended. For example, I am embar- rassed to report that a course in educational measurement principles is not part of the standard teacher education curriculum at the University of Massachusetts (UMASS). Making one or more courses in educational measurement mandatory for teachers and administrators will not only lead to improved measurement practices among these professionals, it will also be an effective recruiting tool. In my experi- ence, when teachers learn about the nuts and bolts of testing, many of them become intrigued. In fact, several of our current and past students were teachers who wandered into one of our introductory measurement courses and never looked back.

When familiarizing others with our profession, we should stress the diversity of our activities. Measurement professionals are often thought of as statisticians with expertise in educational measurement. In this sense, our profession is narrowly viewed as a specialty area of applied mathematics, quantitative psychology, or educational research. However, given the in- creased emphasis on testing in the educational policy community, and in K-12 classrooms, people from less quantitative areas of education are mi- grating toward measurement. Profes- sionals working in a variety of occupations see the need for research and measurement skills. At the same time, they are disillusioned with their other career options. Graduate study in measurement is an attractive place for them to learn these highly coveted skills that are generalizable to a variety of settings.

Reach out to People of Color and Make the Profession More Open and Attractive to Them Relative to other professions in the social sciences, there are few people of color in educational measurement. For some groups, such as African Ameri- cans, the representation is extremely low. There is no reason for such poor representation to continue. Each year at the American Educational Research Association (AERA) conference, I see our brothers and sisters of color everywhere, Everywhere, that is, except at NCME sessions. We need to get serious about the goal of increasing the pres-

ence of people of color in our profession. Seriousness requires fiscal responsibility. The recent Minority Fellowship for Graduate Study in Mea- surement, instituted by NCME, was one step in the right direction. How- ever, the program had little impact in recruiting new professionals, and so it was discontinued. The fact is, NCME does not have the financial resources for serious recruitment of under- represented minorities. For effective recruitment of people of color, financial assistance from major testing companies and the U S . Department of Education is needed. Empowered with a few hundred thousand dollars of scholarship money, we should be able to recruit more people of color into our field. Such financial support could also be used to develop information brochures to be widely disseminated at undergraduate institutions throughout the world, and to pay for measurement professionals to give recruitment speeches at various organizations related to the education of people of color. Perhaps we could hire some people with public relations or mar- keting expertise to assist us with these endeavors. Given the strong job market, the increasingly high stakes associated with the testing of minorities, and improved recruitment efforts, the time is right for us to achieve the goal of a more diverse pool of graduate students and a more diverse NCME membership.

The international students entering graduate programs in the United States and Canada provide another opportunity for increasing the diversity of our membership. Many of the leaders in educational measurement are based in the United States and Canada and the reputation of the United States and Canada for the training of measurement professionals around the world is growing. In fact, many of the leaders in educational measurement were born abroad. As developing countries seek to initiate and improve their educational assessment systems, they will look to countries such as the United States and Canada to acquire their graduate training. Our classes will be enriched by the presence and perspectives of these international students. We must be ready to accept and recruit them. Graduate programs with informative web sites have made it easier for international students to learn about their programs.

Engage Organizations in the Funding and Supporting of Graduate Students As mentioned earlier, there is a great need for measurement expertise. Or- ganizations that need such expertise are increasingly providing internship opportunities to graduate students. The fact of the matter remains there is too much work and too few professionals. By providing internship opportunities, organizations temporarily acquire people with measurement skills and simultaneously woo potential employees. Proactive organizations are not waiting for students to graduate. Edu- cational Testing Service was one of the earliest providers of internship opportunities for psychometric graduate students. Today, American College Testing, the American Institute for Certified Public Accountants, Mi- crosoft, RAND, Professional Examina- tion Service, the National Board of Medical Examiners, and many other organizations are following suit. I ex- pect this trend to continue. These internship opportunities are a win-win situation for graduate students and testing organizations.

In addition to providing internship opportunities, organizations are also contributing to the nurturing of future measurement professionals by award- ing research grants and contracts to graduate programs in educational measurement. A growing number of students have their tuition and sti- pends funded through such research grants, which makes recruitment of students into the field somewhat easier, Such collaborations among governmental, for-profit, and not-for- profit organizations and graduate programs should be encouraged. The aerospace, chemical, agricultural, and many other industries have long sup- ported university-based research. In addition to benefiting the sponsoring organization, such “altruism” is likely to increase the number of educational measurement professionals. If these internships and educational funding grants can be coordinated at a national level, they may be even more effective in recruiting new professionals.

Initiate, Advertise, and Support Nondoctoral Degrees in Educational Measurement The field of educational measurement is extremely heterogeneous. Students who receive doctorates in this area are

Winter 2000 7

typically exposed to a broad range of topics such as test development, classical test theory, item response theory, multidimensional scaling, structural equation modeling, and other multi- variate statistical techniques. How- ever, increasing the number of professionals with measurement expertise does not mean we need to increase the number of professionals with doctorates in psychometrics. Two viable alternatives are to develop and strengthen master’s programs in educational measurement and to offer concentrations in measurement to students in other doctoral programs.

Given the need for measurement expertise, people with master’s degrees in measurement are highly employ- able. In addition, a master’s program provides a more flexible option to potential students who work full time. It is also an effective tool for recruiting future doctoral students. Someone who knows the basic principles of measurement, such as reliability and validity, will be extremely helpful to many testing agencies, as well as to local and state boards of education. A few fundamental measurement courses would make the difference between being a measurement novice and being able to identify and practice sound measurement. Measurement programs could be tailored toward communicating important, but com- plex, testing information to lay audiences, or focus on other specialty areas such as assessment within a program evaluation context. Given the growing interest and activity in public policy regarding testing, and the popu- larity of “nontraditional” assessments, courses in these areas are likely to fill an important need that could help bridge the educational measurement and policy communities (B. Plake, personal communication, May 17, 1999). Providing instruction in these areas may provide an edge when recruiting students to specific programs.

Students in other doctoral programs, such as school counseling, school psychology, educational policy, and educational administration, who receive concentrations in measurement will also increase the number of professionals with measurement expertise. Such professionals will also help to familiarize lay audiences with important educational measurement principles, It is important to note that a population of graduate students with

diverse interests and backgrounds means that a “one-size-fits-all” measurement program may not meet the needs of these various groups. In fact, it may lead to student disillusionment. It may be that some students are taking measurement courses because they want to be educational program evaluators. Other students may want to help local school districts develop district-wide testing programs. Still others may want to work on theoreti- cal developments in the field such as multidimensional item response theory (IRT) models or nonlinear structural equation modeling. Therefore, measurement programs should be flexible in stipulating degree requirements and provide several options to students to best meet their needs. Of course, several courses, such as “principles of testing,” would be requisite.

Develop Distance-Based Training P r o g r a m in Educational Measurement Distance learning and on-thejob training programs represent viable educational alternatives that could be used to train the next generation of measurement professionals. The fast pace of today’s business world and the growing global marketplace will increasingly demand that technical skills be learned quickly, and be transferable throughout the world. It is likely new models of instruction will be developed. Perhaps, as envisioned by Craig Mills (personal communication, March 22, 1999), consortia comprising several universities and testing companies will emerge to provide instruction electronically throughout the world. The relatively new Global In- stitute developed by ETS is an early example of this type of training mechanism. It is difficult to predict the new ways graduate instruction will be delivered in the 21st century, but it is a pretty safe bet to say that it will be different. As Joseph O’Hare, Society of Jesuits (president of Fordham Univer- sity) wrote in a recent alumni news- letter, “one thing we can be sure of in the next millennium is that it will be a time of accelerating change that we cannot predict, much less control” (O’Hare, 1999, p. 2). I think we will still see small groups of doctoral students receiving traditional classroom- based instruction in educational measurement. However, I also think (a) these students will increasingly be involved in applied research projects

early in their graduate studies, (b) distance-based, nondoctoral certification programs in educational measurement will emerge, and (c) more companies will offer on-the-job training in measurement to employees working in applied assessment centers.

Conclusions As I interact with colleagues and students in our field, I am struck by the fact that we all share at least one com- mon attribute: we are all excited about what we are doing. We may be over- worked, but we are working on interesting projects and working with interesting people. As the Emerson quote at the beginning of this paper emphasizes, these are exciting times! A critical goal for us is to share this excitement with those who have the potential to be our future colleagues. I believe we will see an influx of measurement students in our graduate classrooms in the near future. This is not an impressive prediction because it is already starting to happen. How- ever, this positive event will be short- lived, unless we are ready for this influx and adapt our instructional practices accordingly. I believe the law of supply and demand will eventu- ally hold true for the measurement profession. The criticality of measurement expertise is no longer a secret. Discussions of national testing, admis- sions testing, student achievement testing, and teacher testing are taking place in the public forum. For example, Wayne Camara, Director of Re- search for the College Board, recently appeared on Good Morning America to talk about the effects of coaching on the Student Aptitude Test (SAT), and Ronald Hambleton, Head of the psychometrics program at UMASS, recently appeared on the Lehrer News Hour to discuss teacher certification testing. Because of the increasing presence of tests in society, people want to know more about them. This curiosity is providing renewed interest and respect for psychometricians. Many people have treated psychometricians as if they were the entomolo- gists of the social sciences. This image is (thankfully) starting to change. In- terest in psychometrics, coupled with a strong job market for people with measurement expertise, is slowly increasing the number of people who are interested in studying educational measurement at the graduate level.

8 Educational Measurement: Issues and Practice

Given the long and stable trend of too few students in our field, my prediction that there will be an increase in the number of psychometric graduate students is rather bold. However, a key assumption underlying this prediction is that instead of focusing on increasing the number of doctoral degrees awarded in psychometrics, we focus on increasing the number of professionals who possess expertise in psychometric principles and sound testing practices. We need to train professionals interested in educational research and educational policy how to build quality assessments, evaluate assessments, interpret assessment re- sults, etcetera. We do not need to train all people interested in educational measurement how to use item response theory to solve measurement problems. Those of us who enjoy the more technical side of our field can continue to have fun with those important applications. However, the real void, the real need in our profession, is between the extremes of measurement novice and fully trained psychometri- cian. Filling this void presents an exciting opportunity for professors of educational measurement theory and practices, and would help ameliorate the current shortage of measurement professionals.

Notes A previous version of this article was

presented at the annual meeting of the National Council on Measurement in Edu- cation as part of the symposium, What Will Make Educational Measurement Better in the Future?: Our Predictions (John Fre- mer, chair), April 21, 1999, Montreal, Que- bec, Canada. The author thanks John Fremer for coordinating this symposium and for the invitation to participate, Bar- bara Plake for her insightful thoughts on this topic, and Jeff Smith and four anony- mous reviewers for their extremely helpful comments on an earlier draft.

‘Please send your biographies to Ann Wilson, American College Testing, 2201 N. Dodge Street, Iowa City, IA 52243-0618 or [email protected]. Other suggestions for im- proving our recruiting efforts can be sent to me at the address listed on the title page.

References American Federation of Teachers, Na-

tional Council on Measurement in Edu- cation, National Education Association (1990). Standards for teacher competence in the educational assessment of students. Educational Measurement: Issues and Practice, 9 (4), 29-32.

Brennan, R. L., & Plake, B. S. (1991). Sur- veys of programs and employment in educational measurement. Educational Measurement: Issues and Practice, 10

Englehard, G. (197). Introduction. Educa- (21, 32.

tional Measurement: Issues and Prac- tice, 16 (4), 5-7.

Impara, J . C., & Plake, B. S. (1995). Comparing counselors’, school administrators, and teachers’ knowledge in student assessment. Measurement and Evaluation in Counseling and Develop- ment, 28, 78-87.

Kaplan, R. M., & Saccuzzo, D. P. (1982). Pqchological testing: Principles, applications, and issues. Monterey, CA Brooks-Cole Publishing,

Nolen, S. B., Haladyna, T. M., & Haas, N. S. (1992). Uses and abuses of achievement test scores. Educational Measurement: Issues and Practice, 11 (2), 9-15.

O’Hare, J. A. (1999). Meeting the challenges of the new millennium. Fordham Magazine, 32 (3), 2.

O’Sullivan, R. G., & Chalnick, M. K. (1991). Measurement-related course work requirements for teacher certification and recertification. Educational Measure- ment: Issues and Practice, 10 (I), 17-19.

Patelis, T., Kolen, M. J., & Parshall, C. (1997). Surveys of programs and employment in educational measurement, Educational Measurement: Issues and Practice, 16 (3), 25-27.

Popham, W. J. (1991). Appropriateness of teachers’ test-preparation practices. Educational Measurement: Issues and Practice, 10 (4), 12-15.

Schafer, W. D. (1991). Essential assessment skills in professional education of teachers. Educational Measurement: Issues and Practice, 10 ( l ) , 3-6.

Incorporating the Test let Concept in Test Score Analyses Guemin Lee, CTB/McGraw-Hill Robert L. Brennan David A. Frisbie University of Iowa

How should we think about the concept of the testlet? How can testlets be better incorporated into test score analysis? Can there be a one-item testlet?

estlets, as the name implies, have T been defined as smaller subsets of a larger test (Wainer & Kiely, 1987; Wainer & Lewis, 1990). Many widely

used achievement and aptitude tests use testlets to measure various aspects of the construct of interest. Because one of the recent trends in test con-

h e m i n Lee is Research Scientist at CTB/MccGraw-Hill, 20 Ryan Ranch Road, Monterey, CA 93940. His specializations are testlet application, generalizability the0 y, scaling and equating, and imple- mentation of large-scale assessments.

Robert L. Brennan is E.F Lindqubt Professor of Educational Measurement, Lindquist Center, Room 334, Universitg of Iowa, Iowa City, IA 52242. His specializations are generalizabilitg the0 y, classical test the0 y, and equating.

David A. Frisbie is a Professor of Edu- cational Measurement and Statistics at the University of Iowa, Iowa City, IA 52242. His specializations are educational achievement testing and assessment.

Winter 2000 9

recruiting the next generation of measurement professionals

Documents