technical manual - outmatch for...available in the select associate screening system, please see...

Select™ for Leasing Agents (v3)

Technical Manual

Prepared by: Assess Systems

Select for Leasing Agents (v3) Technical Manual

Copyright,1996‐2012Bigby,Havis&Associatesd/b/aAssessSystems.Allrightsreserved. 2

Table of Contents

Overview ....................................................................................................................................................................... 4

General Background ......................................................................................................................................... 4

What Is Select For Leasing Agent (v3) ............................................................................................................. 4

Who Is It For? .................................................................................................................................................. 5

What Does It Measure ...................................................................................................................................... 5

Reading the Report .................................................................................................................................................... 6

What Do The Various Indices Mean? ................................................................................................................. 6

What Are the Select for Leasing Agents (v3) Dimensions? ............................................................................... 7

What Does It Mean When A Performance Subscale Is Flagged? ...................................................................... 9

What Is The Job Task Section? .......................................................................................................................... 9

What Are The Counterproductive Behaviors? ................................................................................................... 10

What Is The Positive Response Pattern? ............................................................................................................. 10

How Do I Use The Interview Probes And/Or Structured Interview? ................................................................. 10

What Is the “Making the Hiring Decision” Chart? ........................................................................................... 12

What Are Development Suggestions? ............................................................................................................... 12

Using the Report ......................................................................................................................................................... 13

Making The Hiring Decision .............................................................................................................................. 13

Selection Recommendations .............................................................................................................................. 13

Who Should See the Select Report .................................................................................................................... 14

Validation Research ................................................................................................................................................... 15

General Background .......................................................................................................................................... 15

Development of Leasing Agent Assessment .................................................................................................... 17

Select for Leasing Agents (v3) .............................................................................................................................. 21

Adverse Impact Analysis with Applicants ............................................................................................................ 22

Adverse Impact Approaches ............................................................................................................................. 22

Adverse Impact for Leasing Agents (v3) ......................................................................................................... 23

Summary ...................................................................................................................................................................... 27

Recommendation of Use ................................................................................................................................... 27



Caution Regarding Interpretation ..................................................................................................................... 27

Next Steps: Monitor Pass Rates ....................................................................................................................... 27

Reference .......................................................................................................................................................... 28

Appendix ...................................................................................................................................................................... 29

Sample Test Items ............................................................................................................................................. 30



Overview

General Background

For over 20 years the organizational psychologists and consultants of Assess Systems™ have been involved in the design, development, and deployment of tailored selection systems for entry-level and para-professional roles across a wide variety of industries. In each case, as part of the project, we were asked to develop and criterion-validate work personality tests to aid in the selection of high potential candidates.

While in each validation study a custom selection tool was developed to meet the unique aspects of a particular job, it became increasingly apparent that these instruments shared many work-personality dimensions. From this research, we developed the foundation for the Select test batteries. Each Select survey measures both those common dimensions predictive of performance in most associate-level jobs, as well as unique dimensions specific to the particular job for which it has been validated. Below is an overview of common dimensions across Select surveys.

Select Dimensions Include:

Positive Attitude - Shows faith in self and positive outlook on self, others, and the company

Energy - Prefers a high activity level; action oriented

Frustration Tolerance - Appears emotionally optimistic and resilient

Acceptance of Diversity - Tolerates others different from self

Self Control - Refrains from impulsive behavior

Acceptance of Structure - Accepts rules and follows direction

Empathy, Interpersonal Insight - Shows an understanding for the feelings and needs of others

Multi-Tasking - Enjoys handling multiple tasks simultaneously

Integrity - Possesses a strong work ethic

Influence - Enjoys influencing the actions of others

Sociability - Displays an ease within a social context and enjoys interacting with others

Good Judgment - Shows a tendency towards objective thinking based on fact

The Select Associate Screening System is a family of related tests designed to measure important, work-related personality characteristics in applicants for entry-level jobs. It allows employers to quickly and easily screen for characteristics associated with productivity, cooperative work behaviors, and integrity. The following pages provide an overview and validation research for the Select for Leasing Agents Survey. For a full listing of other surveys available in the Select Associate Screening System, please see page 34 of this manual.

What Is Select For Leasing Agents (v3)?

Select for Leasing Agents is a personality-based survey designed to measure characteristics that have been found to predict job effectiveness for Leasing Agent positions in the industry.

Through our research with leasing agent jobs, we have identified several key personality characteristics that contribute to job success. Select for Leasing Agents (v3) allows you to hire associates who are positive service providers and be able to influence people to lease or rent apartments. It also helps you to identify those applicants with low integrity and who are poorly suited for these types of roles.



Who Is It For?

The Select for Leasing Agents (v3) Survey has been validated for use with people in Leasing Agent roles.

What Does It Measure?

Standard Modules

Performance and Integrity: Select for Leasing Agents (v3) measures personality characteristics related to effective job performance in administrative or clerical roles. The two central indices are integrity and performance. Each Performance Index has been validated to predict the traits associated with successful performance on the job. The Integrity Index has been validated to predict the attitudes, personal integrity, and work ethic in relation to being effective on the job.

Optional Modules

The following are optional modules offered within the Select system. These modules may be added to your survey or customized to fit your needs. Contact your human resource department or Select distributor for more information.

Job Tasks: The job tasks module is an online interview which asks the candidate about his or her willingness to perform important tasks related to the job. This module includes questions related to effective job performance in administrative or clerical roles.

Counterproductive Behaviors: An additional module available is the Counterproductive Behaviors module (CPB). There are two versions of this module, attitude-based or admissions-based. Both of these are assessments of counterproductive behaviors and include topic areas of drug use, theft, job commitment, work ethic, resistance to direction, safety, etc. Attitude-based questions ask individuals about their beliefs while admissions-based questions ask individuals about prior use (these are relevant to drug use and theft). Caution: Admissions-based questions relating to drug use and theft are prohibited in some states. Check regulations particular to your state laws before implementing.

Leasing Agent Math: The Leasing Agent Math Index is an additional index which indicates the individual's ability to do basic calculations and numerical problem solving typically required for the leasing agent job.



Reading the Report

The Select for Leasing Agents (v3) report provide information to help facilitate the selection of top performing candidates. The reports are designed to provide information regarding a candidate's personality characteristics and highlight areas that may merit attention in the interview process.

What Do the Various Indices Mean?

Random Response Index:

The Select surveys contain a validity check to ensure that the candidate is not responding to the items randomly. Special items in the surveys determine if the applicant is correctly reading the survey and sufficiently attending to the survey items. Inappropriate answers to these statements will result in a survey

which is “invalid.” This may indicate that: 1) the applicant has limited language or reading skills, 2) the applicant did not carefully read the survey, or 3) the applicant may need visual accommodation. We recommend that you attempt to determine the reason for this invalid result and, if necessary, re-survey the applicant providing them with assistance or accommodation.

Integrity Index:

The Integrity Index measures the candidate's attitude related to personal integrity and work ethic. High scores indicate a positive work ethic and low scores indicate a tendency to “get by” and not “give it their all.” While not all applicants with scores in the AVOID range will have low integrity, our research suggests that

many will. Pursuing applicants who score in the GOOD range should increase your chances of hiring someone with a positive work ethic.

Performance Index:

The Performance Index measures those personality characteristics that differentiate good Leasing Agent candidates from poor ones. Those scoring in the higher ranges on the Performance Index (Good and Better) are more likely to possess the important

traits necessary to perform well on the job. Try to avoid hiring applicants with scores in the AVOID range.

Leasing Agent Math:

The Leasing Agent Index measures the candidate’s basic math skills. The index indicates the individual's score within one of two ranges, NEEDS TRAINING or OKAY. It the applicant scores in the NEEDS TRAINING range, he/she may have difficulty, without additional training, performing basic mathematical operations and solving numerical

problems. If the individual scores in the OK range, he/she should be able to calculate rent adjustments, pro-rates, income qualifications and other types of calculations routinely encountered in their jobs.

Random Response Valid Invalid

X

Integrity Index Score:

6

Avoid Good

X

0 - 7 8-13

Performance Index Score:

21

Avoid Okay Good Better

X

0 - 11 12 13-17 18-25

Math Index:

6

Needs Training

Okay

X

0 - 10 11-15



What Are The Select for Leasing Agents (v3) Dimensions? The following table lists personality dimensions found to consistently predict performance in these jobs. These dimensions describe the characteristics which make up the Performance and Integrity Indices.

Select for Leasing Agents (v3) Dimensions

Performance Index Undesirable Scores Desirable Scores Representative

Behaviors

Energy: Activity level, action orientation

Slow pace, low stamina or low energy reserve

Energy, drive to get the job done.

Self-starter

Demonstrate a strong sense of urgency

Effectively manages and coordinates multiple activities

Provides multiple tours and showings daily to prospective residents

Assertiveness:

Can effectively influence customers and take control of

the conversation.

Ill at ease

Reluctant to assert self.

At ease with others.

Direct, will ask for and probe for information.

Finds common ground with customers.

Influences and sells to prospective residents

Assertively close the deal and are not afraid to ask for the deposit

Positive Sales Attitude:

Faith in self, positive outlook on customers and the sales/service

role

Defensive, negative attitude

Sees sales/service roles as subservient to customers

Positive, confident personality projected to customers

Confident in ability to influence results

Trusts in others and the system

Greets prospective residents and listens to their individual wants and/or needs

Answers questions regarding the property, amenities, etc.

Social Comfort:

Comfort in and desire to engage in social interactions.

Social awkwardness or lack of ease in dealing with people.

Dislike of casual social interaction.

Socially comfortable and poised.

Enjoy people and “small talk”.

Makes frequent and positive contact with others

Establishes relationships with prospective residents, renters and owners

Communicates and relate well with others

Maintains relationships with current residents, renters and owners

Accommodation to Others:

Willingness to accommodate the desires of others versus the

tendency to try to dominate.

Tendency to compete with customers.

Tendency to resist unusual requests or special customers’ needs.

Comfortable with the service role.

Has a customer service attitude

Assists residents with property questions and issues/problems

Responds appropriately to resident complaints

Frustration Tolerance: Resilience when faced with frustrations and irritations as

opposed to feeling their effects

Sensitive to frustration Resilient: not easily upset.

Upbeat outlook

Handles difficult conversations with residents in a professional manner

Is resilient

Stays positive and



motivated in the face of difficulty, stress, or rejection

Criticism Tolerance

Can accept critical feedback from others without taking it

personally.

Thin-skinned.

May be overly sensitive or defensive when faced with criticism from others.

Thick-skinned.

Is likely to take criticism from others productively and use it to improve.

Is flexible and adaptable

Accepts feedback in a constructive manner

Self-Reliance:

The ability to work independently with limited

direction from others.

May excessively seek assistance or assurance from others.

May not be confident in self.

Will work independently.

Should complete tasks with little assistance or guidance of others.

Takes initiative

Works with little direction maintaining confidentiality and professionalism

Pays attention to the details and be organized

Stays up to date on competition - shops the competition through phone calls and visits on a regular basis

Acceptance of Diversity:

Tolerance of others different from self.

Impatient.

Annoyed with others who are different from oneself (i.e., foreigners, people poorly informed about products or services).

Cooperative, tolerant with all types of customers.

Treats every resident, potential resident and coworker equally.

Integrity Index Undesirable Scores Desirable Scores Representative Behaviors

Integrity/Work Ethic: Positive outlook on working hard and earning one's pay.

“Get by” attitude; negative attitudes about work and the company

Enjoyment of work; work integrity; honest effort

Good work ethic

Hard worker

Dependable; on time every time



What Does It Mean When A Performance Subscale Is Flagged?

In addition to providing an overall Performance Index score, Select reports also provide subscale information on each of the characteristics that make up the Performance Index. For each subscale, the candidate’s scores will fall into the OK or FLAG category. If a candidate scores unfavorably on a subscale, there will be an “X” next to that subscale under the FLAG heading. For example, if Suzanne Example responded to statements related to Energy in an unfavorable way, there would be an “X” next to Energy on her report. This “Flagged” subscale means that Suzanne may have difficulty maintaining the sense of urgency needed to quickly fulfill customer requests. At the end of the report, suggested interview questions pertaining to the flagged subscales are provided.

It is important to remember that although an applicant’s overall Performance Index score may fall in the “Good/Better” range, he or she may still have a low score on one or a few of the subscales. This indicates that, while he or she may exhibit most of the characteristics identified as important, the applicant may have some weaknesses which can affect overall performance. Any flagged subscales should be followed-up in the interview to confirm whether or not a low score will be problematic.

What Is the Job Task Section? (Optional Module)

The Job Task section is an optional module for the Select surveys. This section asks about the candidate’s willingness to do tasks related to a Leasing Agent position. Areas where the candidate expresses reluctance should be probed further in the interview. (These items can be customized for your organization.)

How willing are you to … Would do it and enjoy it

Would do it

Would do it, but not like it

Would not want to do it

Would not do it

Work evenings or nights? X

Work weekends? X

Work holidays? X

Commit to being on time, every time?

X

Walk around, climb stairs and go outside in any weather?

X

Serve or assist a wide variety of customers?

X

Influence others to make a decision?

X

Work cooperatively with others? X

Do math (add, subtract, multiply and divide) with a

X

Subscale OK Flag

Energy X

Assertiveness X

Positive Sales Attitude X

Social Comfort X

Accommodation to Others X

Frustration Tolerance X

Criticism Tolerance X

Self-Reliance X

Acceptance of Diversity X



calculator?

Operate office machines? X

Answer the phone, respond to requests and provide

information? X

The table above reports the candidate’s stated willingness to do tasks commonly required in jobs similar to this one. Indications of reluctance should be probed during the interview.

What Are Counterproductive Behaviors? (Optional Module)

The Counterproductive Behaviors module will highlight certain items within the Select survey if they are answered in an undesirable way. The items within the attitude-based and admissions-based versions of this module are related to attitudes and behaviors regarding drug use, theft, safety, and similar topic areas. One should use this information to further investigate potential problem areas during the interview or through reference checks.

In this section, undesirable responses by the applicant to drug use, theft, job commitment, work ethic, resistance to direction, etc. questions are presented. The total number of survey questions for each topic is given in parentheses. The candidate selected an undesirable response or answered agree to the following:

Drug Use (5 possible questions)

Which best describes your current involvement with illegal drugs (for example: speed, cocaine, marijuana, heroin, etc.)? Frequent, but I don’t let it interfere with my work.

Using marijuana illegally during work…. Has no effect on me.

What Is The Positive Response Pattern?

Within the Select survey there are statements designed to determine if an individual is trying to present themselves in a more positive manner, or put their “best foot forward.” If these items are answered in an overly positive manner, a special warning will be produced on the report. In these instances, a special effort should be made to verify the results of the report with other information such as interview impressions and reference checks.

Positive Response Pattern

Use Scores Obtained From This Survey Cautiously

This applicant answered the survey in a positive manner. Such positive responses may suggest one or more of the following:

1. He is very positive in his view of himself, others and life in general.

2. He deliberately tried to present himself in a positive manner.

3. He does not have much self-insight and may see few personal weaknesses or areas for self-improvement.

The possibilities listed above should be carefully considered, and a special effort should be made to verify with other information such as additional interview impressions or reference checks.

How Do I Use The Interview Probes And/Or Structured Interview?

The Select report provides interview probes or an interview guide to aid in conducting a structured, behavioral interview that is job relevant. The interview guides were developed specifically for Leasing Agent roles. The guide may be tailored to include your company's custom interview.



Interview Probes

If a candidate flags on any of the Performance subscales, the Select report will include follow-up interview probes. These are presented to assist the interviewer in asking questions about potential liabilities or characteristics which may merit additional attention and consideration. In some instances, an individual may have developed a strategy to compensate for particular weaknesses.

For example, a candidate who scores in the low range on the measure of Frustration Tolerance may become easily discouraged or overly sensitive to job frustrations. However, she may have developed a strategy for using stress management techniques to help maintain her calm and optimism. The interview probes included in the report can help to confirm or deny whether a potential liability exists.

Sample Interview Probe

Low Frustration Tolerance

This candidate scored low on a measure of emotional resilience and positiveness. Some people with this profile can be moody and sensitive, and may be easily discouraged. These characteristics may interfere with her ability to maintain a positive attitude about the job and the people with whom she works.

Interview Probe:

Ask her to describe those aspects of previous jobs which have frustrated or irritated her. Listen for responses that suggest a low frustration tolerance or a tendency to be easily disappointed or upset. Ask how she deals with these.

Attempt to assess her ability to deal with pressure, stress and disappointments on the job. Ask her to describe the sorts of things that irritate or annoy her on the job.

Ask her to describe what she enjoys about working as part of a team and what she dislikes about it. Listen for tendencies to dwell on the negative and, in particular, listen for signs that suggest that her mood is easily affected by things said or done to her.

Probe suggestions allow the interviewer to determine:

To what extent is the characteristic likely to be observed in day-to-day job performance?

What impact (if any) does the characteristic have on job performance, work relationships, problem solving, etc.?

To what extent, or by what method, does the person "manage" or control the behavior?

Structured Interview (Optional)

A detailed interview guide will also be provided in the Select report. This guide will include:

A script to open and close the interview.

A series of job-related interview questions with guidance as to what to listen for in the candidate’s responses.

Interview Probes based on survey results (see above).

Rating Worksheet to help evaluate the candidate’s overall fit for the role.

By using this structured interview guide consistently (meaning you ask the same questions of every candidate), you will bring increased objectivity to the hiring process.



Sample Structured Interview Questions:

While asking the following interview questions, be sure to listen and probe in the following areas highlighted by Select:

Low Energy Level

Low Frustration Tolerance

Tell me about your responsibilities in your previous (or current) job. Which did you like most and why? Also, which did you like least and why? (Listen for relevant work experience and likes and dislikes that may or may not fit this job.)

What were your most significant accomplishments or successes in this (these) jobs? (Listen for work ethic, values, commitment.)

Why are you considering leaving or why did you leave your current/last job? (Listen for reliability, job fit.)

What is the “Making the Hiring Decision” Chart?

The Select report also provides users with a hiring decision chart that identifies key steps in the selection process (e.g. reviewing the application, prescreen, test results, background checks, etc.) and a chart to document notes and overall recommendations. The chart can serve as an efficient way of consolidating notes/observations from the selection process to assist in making the hiring decision.

What Are Development Suggestions?

As an optional resource to assist in on-boarding your new hire, a link for developmental suggestions will be provided at the end of the Select report. Developmental suggestions will only appear for performance sub-scales that are flagged from the assessment. The suggestion will highlight the flagged sub-scale and provide guidance on how to further develop in this area, including on-the-job activities.

Before using these, please contact your organization’s Human Resource department to ensure this is an option your company is using as part of their process. These suggestions should not be provided to applicants who were not selected.

Sample Development Suggestion:

Low Energy/Work Pace

A review of your answers to the assessment suggests that your work style is likely to be methodical and slow-paced. If you need to improve in this area, consider the following suggestions:

Make a “to do” list of your tasks at the beginning of each day. Rank them in order of importance and focus on the highest priority ones first.

Set deadlines for completing a task and stick to these. Sometimes it also helps to make a time schedule for each and every step involved

If you find yourself lacking energy during the day, try to determine why and address the cause: Do you get enough sleep or exercise? Is there something on your mind that is draining your energy?

Don’t jump around among several unfinished tasks. It’s easier to keep your focus when it isn’t divided. Finish the most important tasks and then be purposeful in your intent to complete the rest.



Using the Report

Making the Hiring Decision

As you use Select in your hiring process, please keep the following in mind:

Select results should be one voice in the selection process. Recommendations from Select should be combined with evaluations of work experience and education, and results from interviews with hiring managers to make a final decision.

Select should be an aid to the well-reasoned judgment of the hiring manager, not a replacement for this judgment.

Use the interview suggestions. It may be that a candidate has learned how to manage or compensate for certain weaknesses and using the probes found in the report may help the interviewer to determine whether or not a particular weakness will have an impact on the individual’s job effectiveness.

Selection Recommendations

Using Select for Leasing Agents (v3)

Using an “Avoid the AVOIDs” strategy for both Integrity and Performance, 23.4% of applicants will be screened out as being poor “fits” for the job without introducing significant adverse impact (see Adverse Impact section on page 22). Using this criteria for screening out candidates will reduce interviews with applicants having a poor job fit and will improve your selection decisions.

Should the organization desire to implement a “Raise the Bar” strategy; we strongly advise that an evaluation be conducted on the organization’s applicant population to estimate pass rates and potential adverse impact to ensure that selection strategies meet the Equal Employment Opportunity Commission (EEOC) guidelines. In the general applicant population a “Raise the Bar” strategy, whereby the criteria for screening is set above the Avoid strategy, indicated that Adverse Impact may occur for some protected groups.

The percentage of expected applicants scoring in the recommended ranges for Select for Leasing Agents (v3) is presented in the following table:

Performance Index Range % of Applicant

Sample (n=1446)

Avoid 0-11 23.4

Okay 12 6.2

Good/Better 13-25 70.5

Integrity Index Range % of Applicant

Sample (n=1446)

Avoid 0-7 1.5

Good 8-13 98.5



Who Should See The Select Reports?

Because of the sensitive nature of appraisal information and the dangers of its misuse, the Select report must be kept confidential and its contents restricted to those who have direct responsibility for decision making. The Select reports SHOULD NOT be shown to the individual nor contents discussed with him or her.

The results of Select belong to the hiring organization. The Statement of Informed Consent, designed to notify the candidate of the disclosure policy, is included in both the on-line and paper and pencil surveys. The candidate must read and agree to this Statement of Informed Consent before completing the survey. (Note: When entering paper and pencil results, a Select report should not be processed unless the Statement of Informed Consent has been signed by the applicant -- first page of the survey or answer sheet.)



Validation Research

General Background

The following section will present approaches for developing and validating pre-employment tests and provides a detailed description of the method used in developing and validating the Select surveys. (Additional sections of this manual will present research results for the development and validation of the Select for Leasing Agents survey.) Assess Systems validation research has shown Select traits to predict performance (i.e., the criterion) across a number of customer service, sales, and entry-level manager jobs. There are two main approaches to criterion-related validation studies: predictive and concurrent designs.

Predictive

In a predictive strategy, candidates are tested before they are hired, and performance measures are collected later, after the candidates have been hired and on the job long enough to accurately assess their job performance.

Concurrent

In a concurrent strategy, the relationship between predictor instruments and criterion job performance is verified by testing a sample of current employees. The test performance of these employees is correlated with measures (existing performance evaluations or special purpose ratings) collected “concurrently” or in the same general time period as test performance.

A concurrent validation strategy is most often chosen because of its relative efficiency. It also allows the test to be used for selection sooner. The following flowchart shows the main work steps in a concurrent validation study.

Below are brief definitions of the main work steps:

Job Analysis:

Job analysis is the process whereby the knowledge, skills, abilities, and personality factors (KSAPs) necessary for effective job performance are identified. Through focus groups with job content experts (often supervisors, trainers, and top performing employees), desired attributes, as well as those characteristics that lead to poor performance, are surfaced.

Desired Attribute Dimensions:

Through job analysis, a list (or lists) of the desired KSAPs for a specific position is generated. This list is consolidated and kept as a record of the job relatedness of the developed and validated selection test.

Preliminary Test Battery:

The list of desired attributes helps in the selection of test items and scales which could potentially be used to identify top performing job candidates. These items and scales are the content of the preliminary test battery which is later validated with the incumbent sample.

Job Analysis

Identify Desired Attribute Dimensions

Assemble Preliminary Test Battery

Choose Sample

Administer Test Battery

Collect Job Performance Measures

Statistical Analysis(Validity / Adverse

Impact / Utility)

Final Test Battery



Choose Sample:

In order to “test the test,” a large, representative sample of current employees is asked to complete the preliminary battery. This sample is ideal when made up of both good and poor performers and when there is diversity in terms of ethnicity, age, and gender.

Administer Test Battery:

The preliminary test battery is then given to the incumbent sample. It is important that employees know that they are helping to “test the test” and that none of their responses will be used for anything other than research purposes.

Collect Job Performance Ratings:

Concurrently with administering the test battery, incumbent job performance measures are collected. Specially designed rating forms are used and supervisors are taught how to rate performance as objectively as possible. In some studies, truly objective data can be collected; for example, Select has been validated against individual call revenue in an airline reservations job.

Statistical Analysis:

Analyses are completed to determine the relationship between job performance measures and test results. The primary measure of this relationship is the correlation coefficient. Correlations vary from -1.0 to +1.0.

+1.0 = perfect positive relationship (high test score = high performance.)

0 = no relationship.

-1.0 = perfect negative relationship (high test score = low performance.)

p-value (probability value) indicates level of statistical significance - at a minimum, this should be smaller than .05. (.05 means that the probability of the observed result occurring by chance is 5 out of 100.)

Expected range of correlation coefficients in this type of study: .2 to .4.

After the best predicting scales are combined to form the Performance Index, the indices are analyzed for adverse impact. If adverse impact is found, indices are often modified to ensure the rate of selection does not work to the disadvantage of members of an ethnic, gender, or age group.

Final Test Battery:

Once the best items and scales have been analyzed and selected, a final test battery can be designed. This process generally involves finalizing scoring software and paper and pencil tests.



Development of Leasing Agent Assessment

Development and Validation of Select for Leasing Agents Performance Index

Assess Systems has been working with a variety of property management companies since the mid 1980’s to identify those candidates who will be successful in various job positions within the industry. Assess Systems was asked by a national property management company to develop a survey to predict sales potential and job fit of Leasing Agent applicants.

Through job analysis, a list of attributes desired in Leasing Agent jobs was developed. This list guided the selection and development of a preliminary survey battery. This battery contained scales previously validated in other sales positions as well as experimental scales and items.

In addition, the job analysis provided information to develop the math module. By reviewing training materials, job requirements and interviewing managers and incumbents, the content for the math items was developed.

The final step involved managers reviewing the items to ensure that the types of math problems asked in the survey are actually performed on the job. The math module cut-off score of “Needs Training” was set with a low hurdle of 70% correct.

A representative sample of incumbents participated in a concurrent validation study. Approximately 135 current Leasing Agents completed the survey and at the same time performance ratings were collected from their supervisors. In addition, objective sales data/closing ratios were collected for each employee. So that property location and potential renter volume could be accounted for, each leasing agent’s closing percentage was standardized by dividing the closing percentage by the average closing percentage at each agent’s respective property. After matching survey and performance data and removing individuals with random response patterns, approximately 115 cases remained in the sample.

Various statistical analyses were conducted to determine the best predictor set of test items and scales in the validation sample. The resulting performance (work characteristics) index is significantly correlated with key measures of job performance (see the table below).

Correlations Between Performance Index and Performance Criteria

Performance Criteria Performance

Closing Ratio .41

Overall Work Effectiveness – Supervisor Rating .26

Overall Interpersonal Effectiveness - Supervisor Rating

.27



Leasing Agent Closing Ratio Compared to Location Average

0

20

40

60

80

100

120

Avoid

Okay

Good

The graph below shows the mean difference in closing ratio for each of the Leasing Agent Survey cut-off scores on the Performance Index.



Development and Validation of the Integrity Index

The Integrity Index used in the Leasing Agent surveys was validated using customer service type jobs. A relatively large sample of 416 job incumbents completed the integrity items. The integrity items were developed to assess a candidate’s tendency to “get by” and not to “give it their all” or take advantage of the company. Concurrently, job performance data (supervisor ratings of performance) was collected for each of the participating incumbents; this included ratings of the manager's business ethics.

Statistical analysis, primarily correlation, was conducted to determine the best predicting set of items and scales. This predictor set was then analyzed for adverse impact and subsequently improved to reduce potential bias. The resulting Integrity Index correlated significantly with ratings of business ethics (see the table below).

Correlations Between Integrity Index and Performance Criteria

Job Performance Rating of… Performance

Business Ethics .24

Using supervisor ratings of business ethics, the table below presents average (mean) ratings at various test score ranges. These results show a strong relationship between test scores on the Integrity Index and business ethics.

Mean Rating of Business Ethics by Integrity Index

Integrity Index % of Total Mean

Performance Rating (1-5)

SD F p

Avoid (0-8) 12.5 1.85 .85 5.65 .01

Good (9-11) 30.4 2.04 .95

Better (12-13) 57.1 2.06 .78

Total 100.0 2.01 .81



Cross Validation of the Integrity Index

After initial validation, the Integrity Index was validated in another study of customer related jobs. A sample of recently hired customer service associates served as the validation sample. During their training, the new hires were asked to complete a preliminary survey which included the Integrity Index items. The research nature of the survey was explained and assurances were made that all information would be kept confidential. No one in the organization would see individual survey results. After several months on the job, supervisors completed performance ratings on these associates. The validation sample consisted of 79 associates for whom we had both test data and performance ratings.

As in the initial study, correlations were conducted to determine the usefulness of the Integrity Index. The results of the study indicated that the Integrity Index is a useful predictor of honesty and overall performance. The table below presents the findings of this study.

Correlations Between Integrity Index and Performance Criteria

Job Performance Rating of… Performance

Honesty Behaviors* .39

Summary Service Behavior* .25

* Composite of specific item ratings. All correlations significant at p>.05.



Select for Leasing Agents (v3)

An extensive review was made on the applicant database for Leasing Agents cases gathered to date. Pass rates and adverse impact were analyzed for the survey and when using the “Avoid the AVOIDs” strategy, the survey was within EEOC guidelines. However, recalibrations of cut-scores at the scale and at the Performance and Integrity Index level were now possible to allow for additional selectivity when screening candidates. Below are screenshots of the updated performance and integrity indices.

Select for Leasing Agents version 3.0

Performance Index Score:

12

Avoid Okay Good Better

X

0-11 12 13-17 18-25



Adverse Impact Analysis with Applicants

Adverse Impact Analysis Approaches

Adverse Impact (AI) basically refers to a substantially different rate of selection in hiring, promotion or other employment decisions which works to the disadvantage of members of a race, sex or ethnic group (Uniform Guidelines Questions & Answers #10). Adverse Impact Analysis is particularly important to employers who are governed by the regulations of the Office of Federal Contract Compliance Programs (OFCCP) however it is equally applicable under Equal Employment Opportunity Commission (EEOC) who also investigates claims of discrimination in the workplace.

As specifically defined by Federal Contract Compliance Manual (FCCM), Chapter 7, Adverse Impact is defined as: “Definition: Adverse impact is a substantially different rate of selection in hiring, promotion, transfer, training or in other employment decisions which works to the disadvantage of minorities or women. If such rate is less than 80 percent of the selection rate of the race, sex, or ethnic group with the highest rate of selection, this will generally be regarded as evidence of adverse impact. Adverse impact analyses based on the 80% rule may be buttressed by a test of statistical significance. For further discussion, see Section 7E06.”

Measurement of AI generally falls under two approaches a) evaluation of practical significance or more commonly termed the Four-Fifths Rule as referenced by the OFCCP in their Definition of AI and b) tests of statistical significance.

Practical significance is a legal term that refers to a difference that would be considered substantial by a reasonable person. There is no precise test for practical significance. The 80% Test or Four-Fifths Rule was originally adopted as a test of practical significance. The 80% Test essentially says that adverse impact is suggested if the pass rate of a focus group (e.g. women) is less than 80% of the pass rate of the majority group (e.g. men). Put a different way; is there more than a 20% difference in the pass rates of the two groups? The 80% Test is still used as a useful indicator of practical significance. However, one must look at other factors in combination with the 80% rule – such as the sample size, stability of the trend over time, etc. In some cases, a difference of more or less than 20% may be considered to be insignificant in practice.

Statistical significance refers to a difference that is substantial enough that it is unlikely to have occurred by random chance. A variety of tests exist for examining statistical significance with the two most commonly used being the 2 standard deviation (SD) test, also called the Z test, and Fisher’s exact test (FET). Both approaches examine the relationship between two variables to determine whether a difference in employment decision rates is likely due to chance. Most often the OFCCP has used Z tests (the 2 standard deviation test) as the primary measure of adverse impact particularly for large samples while the Fisher’s exact test (FET) is used in small samples (sample sizes of less than 30).

The reason that the courts have generally applied both statistical and practical significance standards in cases of alleged adverse impact is that statistical significance tests alone can be misleading in some circumstances. In particular, sample size and composition have a substantial impact on tests of statistical significance. A statistically significant finding of adverse impact in a small sample may be suspect because these results tend to be unstable and may not replicate over time. Likewise, in large samples, even very small differences that are insignificant in practice will meet the threshold of statistical significance.

The following Table displays the difference in passing rates, or Impact Ratio, that is necessary to attain statistical significance for a given sample size. The example makes several assumptions (footnoted below) for the sake of simplicity.



Table 1: Minimum Significant Pass Rates*

Sample Size (n) Minimum Significant Pass

Rate Difference Minimum Significant

Impact Ratio

5000 3% 0.97

1000 6% 0.94

500 8% 0.92

100 19% 0.81

50 27% 0.73

* Using Fisher Exact (2-tail), assuming equal division of sample between focal and majority group and a majority pass rate of 75%.

The example in Table 1 highlights two important points. First, even very small differences in pass rates will result in a finding of statistically significant adverse impact when the sample size is large. For instance, with a sample size of 5000, a difference of just 3% will result in a finding of statistical significance even through the impact ratio under the 80% is well within guidelines. Second, even a large difference in pass rates would not result in a finding of statistical significant adverse impact when the sample is small. For instance, with a sample size of 50, a difference of 27% or more is needed to result in a finding of statistical significance. Note that this level of difference would result in a violation of the 80% Test (IR < 0.80) without reaching statistical significance.

As Biddle (2006) points out, “under most circumstances…analysis should result in a firm finding of both statistical and practical significance to constitute a solid finding of adverse impact.” In addition, according to Biddle (2006) “…when practices, procedures, or tests show signs of adverse impact (or may show adverse impact if greater sample sizes were evaluated), the employer should either consider conducting validation studies or using alternative employment practices that have less adverse impact.”

Adverse Impact Analysis for Leasing Agents (v3)

To examine the issue of adverse impact for the Select for Leasing Agents (v3) survey, selection impact ratios for the Performance Index were estimated using a sample of applicant data collected for the standard Select assessment (n= 1446). Recognizing the limitations of using a test of statistical significance on large samples our general approach is to apply the practical significance standard when examining large data sets such as the Select database. Using this approach the hiring recommendations made in the Select Manual will be within EEOC guidelines. (Should an organization desire to use more stringent criteria for screening candidates we recommend that AI be examined at the local level before proceeding.)

Demographic information was collected during the administration of the survey and/or as part of the application process. Ethnicity, age, and gender were examined on candidates that provided the information during the assessment process.

Candidate results were categorized as follows:

Performance Index: Avoid (0-11), Okay (12) and Good/Better (13-25).

Integrity Index: Avoid (0-7) and Good (8-13)



Gender n %

Female 1112 76.9

Male 328 22.7

Prefer Not to Say 6 0.4

Ethnicity n %

African-American/Black (not Hispanic) 362 25.0

Hispanic 251 17.4

Asian/Pacific Islander 25 1.7

American Indian 7 0.5

Native Hawaiian/Pacific Islander 2 0.1

Caucasian/White (not Hispanic) 732 50.6

Other 40 2.8

Prefer Not To Say 27 1.9

Age n %

Under 40 1184 81.9

Over 40 204 14.1

Prefer Not To Say 58 4.0

Adverse Impact Analysis

Analysis was completed on each of the individual indices and scoring recommendations (Performance and Integrity) to review for adverse impact. As can be seen in the upper portion of the table, no significant adverse impact was found for the individual indices (Performance and Integrity) for ethnicity, gender or age.

In addition, analysis of Adverse Impact for the most conservative screening approach was evaluated – that of “Avoiding the Avoids”. “Recommendation 1” in the tables indicates the candidate scored in the Good/Better range on the Performance Index and the Good range on the Integrity Index. Under these criteria 76.6% of the total sample would fall in the Recommend category and no significant adverse impact was found for ethnicity (analysis available for African American and Hispanic – other groups did not have sufficient sample size), gender or age for the combined recommendation.

The values obtained for the test of practical significance are within acceptable ranges based on EEOC guidelines. Organizations could utilize this selection criterion without introducing significant adverse impact on protected groups.



Adverse Impact by Ethnicity (African American, and Hispanic)

Percent of Candidates Passing Select

Index All Applicants

(n=1446)

White

(n=732)

African Amer.

(n=362)

Ratio* Hispanic

(n=251) Ratio*

Performance Index

0-11 Avoid -- -- -- -- -- --

12 Okay 76.6% 81.6% 71.0% .87 72.1% .88

13-25 Good/Better 70.5% 75.4% 63.5% .84 67.3% .89

Integrity

0-7 Avoid -- -- -- -- -- --

8-13 Good 98.5% 99.3% 98.6% .99 95.6% .96

Combined**

Avoid -- -- -- -- -- --

Recommendation 1:

PI: Okay/Good/Better, I: Good 76.6% 81.6 71.0% .87 72.1% .88

Note: Impact Ratios between .80 and 1.20 are considered acceptable under EEOC Guidelines. Recommendation 1 indicates the candidate scored in the Okay/Good/Better range on the Performance Index and the Good range on the Integrity Index.



Adverse Impact by Age and Gender

Percent of Candidates Passing Select

Index All Applicants

(n=1446)

Under 40

(n=1184)

Over 40

(n=204) Ratio*

Male

(n=328)

Female

(n=1112) Ratio*

Performance Index

0-11 Avoid -- -- -- -- -- -- --

12 Okay 76.6% 76.8% 76.5% 1.00 71.3% 78.1% 1.10

13-25 Good/Better 70.5% 70.9% 69.6% .98 65.5% 71.8% 1.10

Integrity

0-7 Avoid -- -- -- -- -- -- --

8-13 Good 98.5% 98.3% 99.5% 1.01 94.8% 99.6% 1.05

Combined**

Avoid -- -- -- -- -- -- --

Recommendation 1:

PI: Okay/Good/Better, I: Good 76.6% 76.8% 76.5% 1.00 71.3% 78.1% 1.10

Note: Impact Ratios between .80 and 1.20 are considered acceptable under EEOC Guidelines. Recommendation 1 indicates the candidate scored in the Okay/Good/Better range on the Performance Index and the Good range on the Integrity Index.



Summary

The following conclusions were drawn based on the creation of the Select for Leasing Agents (v3) assessment:

Those that possess the characteristics measured are rated significantly better in terms of: overall work effectiveness and overall interpersonal effectiveness.

Those that score in the recommended ranges on the assessment are evaluated as better performers.

Recommendation of Use

The Select for Leasing Agents (v3) surveys measure characteristics and skills that have been found to predict job effectiveness for similar types of roles. Through our research, we have identified differences amongst the characteristics that often make an associate more effective in one role versus another.

The Select for Leasing Agents (v3) reports are designed to provide information regarding a candidate’s personality characteristics and highlight areas that may merit attention in the interview process. The following are recommendations on how best to utilize the information from the report:

In prioritizing Leasing Agent candidates, follow the “Avoid the Avoids” strategy; these are candidates who score an “Avoid” on the Performance Index or an “Avoid” on the Integrity index. (This will screen-out an average of 23.4% of Leasing Agent applicants and did not introduce adverse impact in our applicant database. When implementing in your organization, you should continue to monitor screen out rates.)

Should the organization desire to implement a “Raise the Bar” strategy, we strongly advise that an evaluation be conducted on the organization’s applicant population to estimate pass rates and potential adverse impact to ensure that selection strategies meet EEOC guidelines. In the general applicant population a “Raise the Bar” strategy, whereby the criteria for screening is set above the Avoid strategy, indicated that Adverse Impact may occur for some protected groups for the Select for Leasing Agents (v3) survey.

For those candidates who score in the “Good” range, use the assessment results in conjunction with other appropriate information from the application process (e.g., education, prior experience, interview impressions, etc.) to make the final hiring decision.

Utilize the job task and follow-up interview probes to gain further information from the candidate.

Only use the assessment for the appropriate Leasing Agent positions.

Caution Regarding Interpretation

The assessment reports generated by the use of the Select system contain information about individuals that can aid the organization in making selection decisions. Any judgments made in a report are probabilistic only and not deterministic. Ultimate performance on the job is a combination of many factors, including past experience, background, training, organizational conditions, management, etc. The organization must be careful not to overemphasize specific statements or results, but rather consider the total person's suitability for the specific position in the company environment. To minimize chances of erroneous decisions, the organization should integrate the information gained in each report with information from other sources (e.g., interview impressions, references, work experience, job performance, work habits, etc.).

Next Steps: Monitor Pass Rates

Assess Systems will continue to monitor the overall pass rates of applicants tested with the Select for Leasing Agents (v3) surveys and refine and edit as needed to ensure risk for adverse impact is reduced.



Reference

Biddle, Dan (2006). Adverse Impact and Test Validation: A Practitioner’s Guide to Valid and Defensible Employment Testing (Second Edition). Aldershot, Hampshire, London: Gower Publishing.



APPENDIX



Sample Test Items

Sample Select Measures

Personality Scales (sample items):

Counterproductive Behaviors (sample items and responses):



Experience Checklist (items and sample responses from Select for Leasing Agents (v3)) :



Making the Hiring Decision Chart:



Development Suggestions (sample suggestions):



About Assess Systems

Assess Systems is a consulting company that provides products, services and general consulting to organizations.

Our focus is on the people in organizations - which ones to hire or promote and when - how to train and develop them to their fullest potential - and how to organize them and the systems around them to maximize their collective potential.

We are organizational psychologists, management consultants, software designers and service professionals working together to serve the business needs of our clients.

Available Select Surveys The following is a list of surveys available in the Select Associate Screening System:

Select for Administrative Support

Select for Banking Branch Managers

Select for Banking Service Associate

Select for Banking Sales Associate

Select for In-Store Banking Sales Associates

Select for Call Centers - Inbound Combo

Select for Call Centers - Inbound Sales

Select for Call Centers - Inbound Service

Select for Call Centers - Outbound Sales

Select for Call Centers - Sales Combo

Select for Convenience Store Managers

Select for Convenience Store Associates

Select for Customer Service

Select for Entry-Level Retail Managers

Select for Health Care

Select for Help Desk Agents

Select for Hospitality

Select for Leasing Agents

Select for Personal Service

Select for Production & Distribution

Select for Retail Clerks/Cashiers

Select for Retail Sales Associates

technical manual - outmatch for...available in the select associate screening system, please see...

Documents