Download - Using tests to predict job performance CALSWEC - 2007

Using tests to predict job performanceCALSWEC - 2007

Cynthia Parry, PhDC.F. Parry Associates

Michelle Graef, PhDCenter on Children, Families & the Law –

University of Nebraska, Lincoln

Todd Franke, PhDDepartment of Social Work –

University of California at Los Angeles

Henry Ilian, PhDNew York City Administration for Children’s Services

James Satterwhite Academy for Child Welfare Training


• Goals:– Define high-stakes testing– Review technical complexities and professional

standards– Discuss the risks, rewards & consequences of

high-stakes testing

• Anticipated Outcome– A phone call to one or more of the presenters prior

any venture in high-stakes testing development


Presentation in 4 parts:1. Overview of high stakes testing

2. Validity considerations

3. Validity issues in high stakes testing and training of child welfare workers

4. The ethics of using high-stakes testing and the ethics of not using high stakes testing

Part 1 of 4:Overview of High Stakes Testing

Cindy Parry, PhD

What do we mean by high stakes?

• Any use of a test that might affect– Your ability to get or keep a job– Your chances for promotion or raises– Your supervisor’s evaluation of your work– Testing out of training requirements or needing further

coaching or other interventions• Such as

– Licensing exams– Civil service hiring tests– Certification/credentialing tests– Any testing where individual results are reported to a

supervisor/manager or used to predict future competent performance

What is the Law and Policy Context?

• 14th Amendment Requirements– Equal protection– Due process

• Civil Rights Acts– Disparate treatment of protected group– Disparate impact on a protected group– Business necessity

• ADA– Accommodations for test takers

Taken from Mehrens, W. Legal and Professional Bases for Licensure Testing. In Impara, James (Ed.). (1995) Licensure Testing: Purposes, Procedures and Practices. Buros Institute of Mental Measurements: University of Nebraska, Lincoln.

Considerations in Developing a High Stakes Test

• Use professional standards to guide test development – The Standards for Educational and Psychological

Testing (AERA, APA, NCME) – Uniform Guidelines on Employee Selection

Procedures (EEOC)– Principles for the Validation and Use of Personnel

Selection Procedures (SIOP)

Key Considerations in Validation

• Validity is the most fundamental consideration in developing any test

• We validate the use of a test score for a specific purpose — not a test or an item

• Content validity is the most common type of evidence cited in child welfare applications– Requires full specification of the domain or construct it is

intended to measure (1.6)– Standards require complete description of procedures used

to develop test content (1.6)• Use of subscales requires validation of the inferences

made from subscale scores as well as overall score (1.12)

Standards cited from: American Educational Research Association, American Psychological Association and National Council on Measurement in Education. (2002). Standards for Educational and Psychological Testing. American Educational Research Association: Washington, DC.

Other Considerations for High Stakes Testing

• Adequate reliability for overall scores and any sub-scores that will be used in decision making

• Investigations of differential item functioning and test fairness

• Comparability of alternate forms or tests given over time

• Documentation of the method of setting the cut score and the qualifications of judges

• Notice of testing and due process procedures• Policy and procedures for accommodating persons

with disabilities • Policy and procedures for test security and

identification of test takers

Part 2 of 4:Validity Considerations

Michelle Graef, PhD

Can we use our training evaluation knowledge test to make decisions about candidates’ ability to

perform the job?Three points to consider:• Used alone, a written knowledge test is unlikely to be

adequate to make inferences about the full range of KSAOs needed to perform child welfare work.

• The target of your inferences indicates the appropriate content domain to be sampled. The typical training evaluation knowledge test (“post-test”) items have been sampled from the curriculum domain, which may not accurately represent the job domain.

• Even if you have strong evidence of content validity, evidence derived from other validation strategies (criterion-related validity) is recommended to support an employment decision. The implied inference is prediction of future job performance.

Job Domain (KSAOs): Determined through a Job Analysis

Knowledge

Skills

Other Characteristics (values, beliefs, attitudes, personality)

Abilities

Training Curriculum Content [K,S,O]

Test Item Content [K]

Sampling for development of a training evaluation knowledge test designed to assess trainees’ “mastery of training content”




Job Domain [K,S,A,O]

Example of a training evaluation knowledge test that is inappropriate for assessing individuals’ “ability to perform the job”


Test Item Content [K,S,O]

Job Content [K,S,A,O]

IDEAL: Appropriate sampling for training evaluation test designed to assess individuals’ “ability to perform the job”

Predictor Measure (e.g., test)

Criterion Measure (e.g., supervisor rating, case review scores)

Criterion-related validity

Predictor Construct Domain (e.g., knowledge of child maltreatment, critical thinking skill, conscientiousness)

Criterion Construct Domain (e.g., job performance)

Content validityConstruct validity

Adapted from: Standards for Educational & Psychological Testing (1999)

Conceptual basis for test validation strategies

Criterion-related validation study

• Demonstrate relationship between predictor and criterion using statistical significance testing

• Feasibility depends upon:– Availability of appropriate criterion measures– Representativeness of research sample– Adequacy of statistical power

• Variety of designs:– Predictive– Concurrent– Use of incumbents or job applicants

• Development of predictor and criterion measures that are relevant, uncontaminated, not deficient, free from bias, and demonstrate reliability

• Refer to Principles for the Validation and Use of Personnel Selection Procedures (2003)

Recommendations

Strategies:

1) retrofit: map your existing knowledge test to the job domain and supplement it with other assessments (e.g., skills) to more adequately represent the job domain

2) start over: develop a test specifically for this purpose that directly samples the job domain, such as a work sample test, simulation, or assessment center

3) gather evidence of criterion-related validity in support of the use of your existing training evaluation knowledge test for employment decisions

Part 3 of 4:Validity issues in high stakes testing and training of child

welfare workers

Todd Franke, PhD

Validity issues in high stakes testing and training of child welfare workers

Are we measuring what is important?

Los Angeles Context

Evidence based practice

Evidence based training

High stakes in Los Angeles

Minimally competent CPS workersWhat would one look like?

Translation validity -- Focuses on whether the operationalization (i.e., measure) is a good translation of the construct

Face Validity -- On its face, does the operationalization look like a good translation of the construct?

Content Validity -- Operationalization is checked against the relevant content domain for the construct

Criterion-related Validity

Predictive Validity -- Operationalization’s ability to predict something it should theoretically be able to predict (L3)

Concurrent Validity

Convergent Validity

Discriminant Validity

Construct Validity

So what constitutes the domains relevant in training child welfare workers?What should be trained/assessed?

It depends who you ask!

Knowledge of

Policies ( over 384 policies in LA-DCFS)

Procedures

submitting mileage

cellular phone reimbursement

use of logs (court calendars, visitation)

new case to-do’s

cubicle organization and systems

Examples of current topics

Appreciating multiculturalism

Family preservation/Alternative response

Adoption assessments

Home assessments

Team Decision Making

Going to court/testifying

Court report writing

SDM tools

Basic interviewing

Adult substance abuse

Child neglect

Concurrent planning

Legal foundations

Risk assessment

Sexual abuse

Legal foundations

Legal permanence

Social work values

Worker safety

Physical abuse

What else might be important?Suggestions from supervisors/trainees

Readiness to learn

Ability to multi-task

Willingness to accept supervision

Ability to make hard decisions

Ability to transfer knowledge to field

Social support

Problem solving

Are these training issues or hiring criteria?

Writing

Self-reliance

How to deal with emotional aspect of job

Critical thinking

Issues related to burn-out

Anything that links directly to case work practice

What’s next?

How should an assessment be used?

Is knowledge only one of the domains that define the construct?

Does it have any predictive validity?Is it a valid measure of future job performance?

Part 4 of 4:The Ethics of Using High-Stakes

Testing and the Ethics of Not Using High Stakes Testing

Henry Ilian, PhD

High-stakes Tests Accomplish Three Things

• Screen out trainees who cannot demonstrate a specified level of mastery

• Compel Studying

• Enforce Fidelity to the Curriculum

Compel Studying N Items Pretest Posttest

CW Common Core 74 80 39.70% 86.10%

(High Stakes)

Core Essentials for 64 32 44.30% 63.40%

Experienced CWs (Not High Stakes)

Enforce Fidelity to the CurriculumCore Essentials for Experienced CWs

Posttest 1 64 15 72.80%

Posttest 2 64 17 55.10%

A Comparison from NYC

Using and Not Using High-Stakes Tests Each Has Consequences

• Using High-Stakes Tests– The requirements for a professionally developed

testing program require significant agency resources

– Some people with the potential to be good child protective workers may not pass

• Not Using High-Stakes Tests– Risk to families, children, self and colleagues– Difficulty maintaining employee standards and

agency morale

The Major Ethical Issue is the Potential for Harm

• There is a need to balance potential harm – to caseworker trainees – to children and families served by CPS

agencies – to co-workers and the agencies

themselves

Concerns for the Evaluator

• The organization may be reluctant/unable to make available the time and resources to develop and validate instruments

• Testing decisions are often made in a political or administrative context

• Administratively imposed testing may not meet professional standards

Download - Using tests to predict job performance CALSWEC - 2007

Top Related