IMPACT HIRING:

HOW DATA WILL TRANSFORM YOUTH EMPLOYMENT

Niko Canner, Incandescent

Abigail Carlton, The Rockefeller Foundation

Guy Halfteck, Knack

John Irons, The Rockefeller Foundation

EXECUTIVE SUMMARY

TWO PAIRED CHALLENGES:

Youth Employment and Talent

Quality at the Entry Level

OUR STUDY:

Predicting Job-Specific Performance

UNDERSTANDING THE EVIDENCE:

Debunking Six Myths

THE DAMAGE THE MYTHS DO:

Systemic Failures in Talent Markets

A PRESCRIPTION:

Data-Driven Hiring for Entry-Level Jobs

APPENDIX

ACKNOWLEDGEMENTS

CONTENTS

0201

Existing programs to address this challenge are expensive and difficult to scale, underscoring the need for a radically different approach to effect change at the ecosystem level.

Advances in predictive talent analytics form an important element of a scalable solution. These technologies enable employers to identify promising talent from a larger pool than they might traditionally consider and to make hiring decisions based on data rather than intuition. Because predictive talent analytics enable two-sided matching of youth with opportunities based on potential, youth seeking jobs can identify the opportunities that best match their talents, while employers can identify jobseekers who are best suited to perform successfully in their specific open roles. This creates a more liquid, better-functioning talent market, which breaks down a number of the barriers that hold youth back today.

To the best of our knowledge, the promise of these new talent analytics tools has not been tested in the context of youth employment. The Rockefeller Foundation and its grantee

EXECUTIVE SUMMARY

Youth unemployment in the United States is an intractable problem, especially for the vulnerable segment of “opportunity youth”: young people between the ages of 16 and 24 who are neither in school nor work.

Incandescent therefore partnered with Knack, a startup company that has developed mobile video game technology to measure talent markers and predict job performance potential, to conduct a pilot of game-based talent analytics with youth enrolled in programs at a number of community organizations. Six hundred “opportunity youth” participated in the pilot, allowing us to compare the full data set of their aptitudes revealed through gameplay with the corresponding data set for current jobholders at four companies.

The results of our study are compelling. Among the six hundred youth, 83% scored at or above the level of the company’s average performers for one or more jobs. This research breaks down prevalent (and costly) myths about opportunity youth, presenting evidence that they have a similar distribution of traits and aptitudes as the general population, that they have the high potential necessary to advance beyond entry-level jobs, and that the aptitudes and skills they demonstrate are not related to gender, ethnicity or level of educational attainment.

As so many fields have in recent years, entry-level hiring must also make the transition from relying on untested intuition to leveraging the power of data and evidence. Employers now have access to talent analytics tools that can enable them to develop a deep understanding of what attributes drive good performance for their current employees, apply tools to objectively assess these attributes, and access broader talent pools to find individuals with the most-valued attributes. The talent analytics tools that enable this vision for data-driven hiring already exist. The key obstacle to their implementation is institutional will.

Our study strongly indicates that data-driven tools hold great promise for changing the course of youth employment. Abandoning misconceptions about opportunity youth and transforming outdated hiring practices will not only enable employers to source talent widely and build a high quality entry-level workforce, but will also move the needle on youth employment at the ecosystem level.

04

TWO PAIRED CHALLENGES: Youth Employment and Talent Quality at the Entry Level

Intractably high youth unemployment in the United States poses a threat both to our young people and to our national prosperity.

Following the Great Recession, youth unemployment reached a 50-year high before leveling off at 12.2%, more than double the general unemployment rate of 5.5%.1 The costs of gaps in early employment for the more than 5.5 million young adults out of work are immense—the total social cost, including factors like lost earnings, lower economic growth and higher government spending, has been estimated at $4.75 trillion in present value terms. 2

Within the broad range of unemployed youth there is a particularly vulnerable segment: those between the ages of 16 and 24 who are neither working nor enrolled in school. Often referred to as opportunity youth, these young Americans frequently face additional life challenges, including lack of family and social support and severely limited access to quality education. Young people who are unable to find stable jobs by their early 20s are at risk of more frequent and prolonged spells of joblessness, permanently lower earnings over their lifetimes, and greater difficulty building a secure financial future for themselves and their families.

A wide array of programs exist to prepare opportunity youth to gain access to employment and succeed in their early roles. While they can be effective for many young people, the best of these programs are quite capital-intensive. YearUp, for instance, is an intensive one-year program that provides low-income young adults with both hands-on, job-specific training and a 6 month job placement at a tech or finance company—at a cost of $25,000 per participant per year.3 Addressing the challenge of youth unemployment to any significant degree requires a dramatically less expensive approach that can be scaled up much more rapidly.

At the same time that youth unemployment remains stubbornly high, many employers still struggle to find high quality talent for entry-level roles. Because entry-level employees represent a relatively large cohort in any business and are often the front-line workers who interact with customers, any improvement in job performance translates into significant business value and cost savings. In call centers, for example, higher-performing operators resolve more problems

on the first call, generate higher customer satisfaction and complete more work per hour. In retail, food service and other fields, front-line workers are the face of the brand, significantly impact critical variables such as speed of service and contribute to customer loyalty and sales. Across a wide spectrum of roles, hires with a “better fit” stay longer, enabling their employers to avoid the costs of fluctuating staffing levels and of recruiting and training replacements.

The Rockefeller Foundation has engaged with employers to explore the concept of “impact hiring”—an innovative approach that enables employers to find better, more successful matches for entry-level positions by drawing on the youth talent pool. Impact hiring means rethinking how talent is sourced, measured, matched, and supported, implementing new processes and tools to improve entry-level hiring across employers. This study explores how employers can realize economic value by adopting novel tools and practices that efficiently unlock access to talented opportunity youth.

1. “The Employment Situation.” U.S. Department of Labor, Bureau of Labor Statistics, June 2015.

2. Bielfeld, Clive R., Henry M. Levin, Rachel Rosen, 2012. “The Economic Value of Opportunity Youth.” Washington, D.C.: The Corporation for National and Community Service and the White House Council for Community Solutions.

3. Roder, Anne and Mark Elliott. 2014. “Sustained Gains: Year Up’s Continued Impact on Young Adults’ Earnings.” New York: Economic Mobility Corporation.

03

06

OUR STUDY: Predicting Job-Specific Performance

Knack’s game-based technology has been used to predict on-the-job performance in a wide range of applications—from finding entry-level workers for retail, operations, sales, and business management roles, to identifying high-potential scientists for Shell’s R&D Innovation program and surgery residents for NYU Medical Center. Working with 18 youth service providers that serve low-income young adults not currently employed or enrolled in school, Incandescent enlisted over six hundred participants, ages 18 to 24, to play the Knack mobile games. The results were striking: among this group, large numbers of individuals had the attributes that differentiate high performers from low performers in each of a spectrum of entry-level roles.

Game-based talent analytics can do what is all too rare in the recruiting field: establish a quantifiable predictor of workplace performance that can substitute for the use of traditional qualifications in screening and hiring. Knack’s games are designed to elicit a wide variety of individual responses to different situations, which are mapped to a range of cognitive abilities, personality traits, emotional and social abilities, mindsets and aptitudes. By using independent, validated measures of what they call knacks (e.g., logical reasoning, social intelligence, motivation, resilience, creativity, action orientation), Knack builds models that turn gameplay data into trait and ability scores.

With these trait and ability scores, Knack then builds models to predict job performance in a broad range of occupations. One of the ways in which Knack builds these models is through working directly with employers: employees play the games, then Knack uses the data from a part of the employee sample to develop a custom predictive model. This model correlates gameplay data with the company’s own performance ratings (e.g., an overall performance rating or specific measures of productivity). Then the model is tested against the rest of the employee sample to evaluate its power to predict performance. These predictive models, called ultraknacks, are used to assign any individual who plays

To test this proposition, The Rockefeller Foundation and grantee Incandescent, a strategic consultancy, worked with Knack, an innovator in the predictivetalent analytics space.

the games a score that predicts their level of performance in the role, based on the specific footprint of traits and abilities that individual demonstrates through gameplay.

Among our sample of six hundred youth, we used Knack to score youth in terms of their potential to perform successfully in four separate jobs: (1) entry-level customer service role at a financial institution, (2) claims processing role, (3) restaurant service role in a restaurant chain, and (4) highly skilled financial analyst role in a large insurance firm. In each case, Knack had already developed the ultraknacks for these roles using the employer’s own workforce, so we were able to identify youth talent using these validated predictive models.

The Knack scores should be viewed as indicative of the potential to perform successfully on the job, all else being equal, rather than as an estimate of what the youth’s performance would actually be. For instance, all the individuals in the financial analyst role have training and experience in finance. One would expect that anyone who entered into this job without that training and experience would struggle to deliver on the basic requirements, regardless of the individual’s other aptitudes. However, Knack’s predictive models show us how an individual would be most likely to perform with equivalent training.

05

07

UNDERSTANDING THE EVIDENCE: Debunking Six Myths

REALITY: A large number of “opportunity youth” off the radar of most employers demonstrate the quantifiable attributes predictive of success in high-quality entry-level jobs. Many of them outscore the individuals working in these jobs today.

This reality debunks a first commonly-held myth: that only a few “opportunity youth” have what it takes to outperform the more experienced and educated people companies are recruiting into high-quality entry-level roles. The customer service ultraknack we used in this study was built using data from employees in a customer service role in a large financial institution—a high-quality entry-level job in terms of skill level, working conditions, pay and potential for advancement. As a predictive model, this ultraknack demonstrates statistical significance far beyond the 5% threshold. In contrast to most of the subjective data, like interviews and resume reviews used by most employers to guide hiring decisions, this model produces a clear, quantifiable and pronounced difference between higher and lower job performers.

Our study punctures six myths about youth and employment.

08

> Avg performers

> High performers

Customer Service Ultraknack Score

39% 28%

Employee distributionYouth distributionYouth > avgYouth > high performer avg

> Avg performers

> High performers

Finance Ultraknack Score

32% 11%

Employee distributionYouth distributionYouth > avgYouth > high performer avg

1009

Applying the customer service ultraknack to the six hundred youth in our study showed that:

• 39% score higher than the average of current customer service employees at the company

• 28% score higher than the average of current high-performing employees in this role at the company

In other words, two in five of the youth in these eighteen programs, which focus specifically on young people who face significant obstacles to gaining employment, score higher on Knack’s predictive model than the average employee already working in this high-quality entry-level job—and one in four perform better on the predictive model than the employees who have been identified as the company’s top performers.

This striking result was not limited to the customer service role. In fact, the data showed similar results for the three other roles we included in this study. Specifically, 32% of the youth in our sample score above the average among high-performers in an entry-level claims processing role at a large employer in this space, and 29% score higher than the average among high-performers in a restaurant service role at a California restaurant chain operator. Even for a highly-skilled entry-level financial analyst role at a global health insurer, 32% of the youth in our sample score at the level of the average current employees in this role, and 11% score as high as the top quintile of star financial analysts working at this firm.

Figures 1 and 2: Substantial portions of the youth population show high performance potential

Youth (blue line) and employee (black line) distributions of scores on the Customer Service

Ultraknack (top) and Financial Analyst Ultraknack (bottom). Shaded areas show youth population

scoring above the average for all employees (light blue) and above the average for high-

performing employees (dark blue).

REALITY: “Opportunity youth” have a distribution of traits, abilities and aptitudes no different from that of the general population.

This debunks a second commonly held myth: that the aptitudes of “opportunity youth” differ systematically and significantly from those of the broader population.

Take a moment to review the six alphabetically listed knacks below. For three of these knacks, the six hundred youth in our sample were at least 10% more likely than the general population to score above a threshold indicating an area of strength. And for the other three knacks, these six hundred youth were at least 15% less likely than the general population to score above that threshold.

Digital MindsetPlanningResilienceSeeking AdventureStaying FocusedTeamwork

Youth lower score probability

Youth higher score probability

Distribution overlap

Youth distribution

General population distribution

Digital Mindset Knack Score People Person Knack Score

Teamwork Knack Score

1211

There are many arguments out there suggesting that youth bring particular strengths to the workforce that employers should seek out. Digital Mindset, Resilience, and Seeking Adventure echo themes one frequently hears in these kinds of discussions. Youth can bring tech savvy, a sense of daring to try new things, and the resilience that comes from tackling hard life circumstances. In our experiment, however, these are three knacks on which our opportunity youth are somewhat less likely to excel than the general population. Planning, teamwork and staying focused are all areas in which these youth are more likely to excel than the general population, dismantling stereotypes that youth today are more easily distracted, less disciplined and more in it for themselves than their older counterparts.

While these six results are all above the threshold of statistical significance, the fact remains that the similarities between youth and the general population far outweigh the differences. Figure 3 provides a visual of this similarity. When looking at how the youth and general population score on different knacks, the overwhelming similarity is readily apparent by the large amount of overlap represented by blue shading. If the data had supported the myth that youth differ significantly and systematically from the general population, we would expect to see far less overlap in the distributions of scores on far more knacks.

What matters from an employment perspective isn’t that youth, as a cohort, are different from other age groups in the talent market. What

Figure 3. The youth sample is broadly similar to the general population

Youth (blue line) and general population (black line) distributions of scores on the Digital Mindset

knack (top left), People Person knack (top right), and Teamwork knack (bottom left). Shaded

areas indicate overlap of youth and general population scores (blue), where youth score below the

general population (red), or where youth score above the general population (green).

matters is that youth are different from one another, and have a wide range of aptitudes that suit different individuals for distinct types of jobs.

REALITY: “Employability” isn’t an assessment that can be applied across the board to whole cohorts of youth. Different youth excel at the aptitudes that predict success in different roles, and most of the youth in our sample fit the success profile for at least one of the four target roles well.

This debunks a third commonly-held myth: that some youth are simply more employable than others, and have greater aptitude across the spectrum of entry-level roles.

One might intuitively picture “employability” as a kind of cocktail of attributes that would apply in very similar ways to different kinds of entry-level work. For instance, many people believe a combination of ability to understand people, self-discipline and communications—or some set of traits like these—form a “soft skills” core that determines whether people will perform well, acceptably or poorly in any of the four entry-level jobs we focus on in this study. According to that view, if Knack or another talent analytics company is able to predict who will perform how well, one would assume they must be testing for something like that core group of soft skills. This view implies an expectation that youth who score high on the core group of soft skills, as measured by the games, would be predicted to excel across a range of jobs—and those youth who score low on this core skill set would be predicted to perform poorly across a similarly wide range of

jobs that have similar determinants of success.

Intuitive as such a picture might seem, it is absolutely at odds with the data from our study. Even looking at just four jobs, we found:

• Only 6% of the youth were “all-around super athletes” who scored at least at the average of the company’s designated high performers for at least three of the four jobs…

• … but 65% of the youth scored at least at the average of the company’s designated high performer for one or more of the four jobs…

• … and, as is illustrated in Figure 4, 83% scored at least at the level of the company’s average performers among their current employee base for at least one job

If one expanded the range and diversity of the jobs considered, one would undoubtedly see an even stronger version of this result. The aptitudes relevant for work in construction, for example, would likely not match particularly closely the aptitudes relevant for restaurant service or claims processing.

The reason we see this result is that different knacks are in fact positively—and negatively—associated with success in these four jobs.For example, as detailed in Figure 5, high-performing customer service representatives show strong resourcefulness, self-confidence, and inspirational leadership abilities while also being more introverted and spontaneous; whereas high-performing financial analysts are optimistic, pragmatic, risk averse, and able to regulate their negative reactions. While a tiny number of people might embody both profiles, these two jobs in fact require attributes that pull in different directions. As a larger number of employers get more granular and

No match

Customer Service

Claims Processor Restaurant Servicer

Financial Analyst

# matches n (%)

Generalists

Specialists

0

4321

99

791

211179

(17%)

(1%)(16%)(36%)(30%)

Figure X. The knack pattern for each role varies based on abilities and traits needed for high job performance. Knacks positively correlated (top) and negatively correlated (bottom) with the ultraknacks used in the study are shown for two roles: customer service (left) and financial analyst (right).

Customer Service Financial Analyst

Positive Correlation

Negative Correlation

1413

more accurate about exactly what attributes matter, the net effect is that a wider range of candidates shine to employers who value their particular, distinctive mix of abilities.

This is a deeply encouraging result for the field of youth employment. If employability were more single-faceted and broadly consistent across most entry-level jobs, it would be difficult and expensive to move the needle for those youth who are currently less employable. The Knack data indicate that employability is a not a single thing, but a diverse, heterogeneous set of factors that vary significantly across different entry-level jobs. This points to an opportunity to strengthen the focus of the youth employment field on matching: helping employers identify youth who already have the aptitudes associated with success in a particular job, and enabling youth to identify opportunities that are the best match for them.Figure 4. There are well-matched opportunities for most youth

Number of youth (each dot is 1 individual) shown matched to different subsets of the four

jobs (ovals) based on the average of current employees: Customer Service (brown), Claims

Processing (red), Restaurant Servicer (blue), and Financial Analyst (green). Youth matched to

one job (white circles) are shown at the outer ends of the ovals, whereas youth matching to

two (large gray circles), three (small gray circles), or four (black circles) job are shown towards

the middle of the figure in the appropriate overlapping segments. Youth not matching any

jobs (x’s) are shown outside the ovals.

Figure 5. The knack pattern for each role varies based on abilities and traits needed for high job performance.

Knacks positively correlated (top) and negatively correlated (bottom) with the ultraknacks

used in the study are shown for two roles: customer service (left) and financial analyst (right).

REALITY: While skills and knowledge gained in the classroom may be directly relevant for certain jobs, education is not a proxy for cognitive abilities relevant to job performance.

This debunks a fourth commonly-held myth: that education can be used as a proxy for a job candidate’s ability to perform.

Employers routinely use education as a proxy for performance when making hiring decisions. They may, for instance, not care particularly about any specific material that a college graduate learns in the university classroom,

but believe that college graduates have certain general abilities that will help them to succeed: better problem-solving skills, perhaps, or better ability to use data to inform decisions.

These generalizations can be hazardous to both employers and job seekers. Research conducted by Innovate+Educate, an organization that employs research-based strategies to address the national “skills gap,” found that while only 1% of unemployed New Mexican young adults meet the criteria for jobs that require a college degree, 33% meet the qualifications when their aptitude is instead measured by skills. Employers in New Mexico who partnered with Innovate+Educate to leverage a skills-based hiring approach experienced 25% - 75% reductions in turnover, time to hire, cost to hire and a reduction in training.4

Similarly, our study of youth underscores the lack of close connection between educational attainment and the aptitudes that distinguish high performers in entry-level roles. Given the nature of our sample, we could not contrast college graduates with high school graduates, but we were able to compare the scores of the 155 youth with some college to those of the 430 youth with no college. In no case was the percentage of youth scoring above the average of the employees currently in one of our four roles, or above the average of the employer’s designated high performers, significantly different between the “some college” and “no college” sub-groups in the sample.

Take, for instance, the financial analyst role—the role one would most naturally associate with the kinds of aptitudes that might be prevalent in a college-educated population, and for which an undergraduate degree is indeed a prerequisite. 34% of the “no college” youth score at least as high as the average of the employees currently in the analyst role,

4. Blivin, Jamai and Merrilea Mayo, 2013. The U.S. Skills Gap (White paper) Innovate+Educate.

0%

10%

20%

30%

40%

50%

60%

Perc

ent o

f you

th e

xcee

ding

thre

shol

d

CustomerService

ClaimsProcessor

RestaurantServicer

FinancialAnalyst

Some CollegeNo College

1615

versus 29% of the youth with some college, a difference that does not reach the five percent threshold of statistical significance. 11% of both groups score at the same level as those analysts the employer called out as the highest performers.

This result does not preclude the possibility that education could be associated with superior performance on the job in the kinds of roles we used in our study. For instance, completing an undergraduate major in accounting almost certainly confers skills and knowledge to some degree relevant to success in the financial analyst role. Knack’s predictive talent analytics wouldn’t pick up this effect, both because all

Figure 6. Education level makes for a poor proxy of performance potential

Percent of youth with some college education (green) or no college education (orange)

whose ultraknack scores exceed the average performance of employees in each of the four

jobs. The lighter-shaded areas of the bars indicate the percent of youth whose ultraknack

scores also exceed the average of high-performing employees.

of the financial analyst employees who played the games had a college education and because Knack’s games don’t measure accounting expertise.

The trap to avoid, which these results underscore, is looking at education not in terms of its direct relevance to the work, but as an indirect proxy for other aptitudes. By identifying and measuring aptitudes more directly, we can begin to see “hidden talents” that employers could easily miss as they implement protocols for resume screening that build in incorrect, non-empirical assumptions about what education or experiences are proxies for what desired qualities.

REALITY: “Opportunity youth” score as strongly on attributes associated with the potential to advance into higher-level roles as other demographic groups. “Potential” is a useful notion only when it can be translated into specific attributes that can be measured; once this translation is made, youth are as likely to excel at those attributes as others.

This debunks a fifth commonly-held myth: that youth may be able to perform well in entry-level roles, but many do not have the aptitudes associated with advancement into higher-level roles.

The best employers focus not only on hiring individuals able to perform in the immediate job at hand, but on hiring and nurturing individuals who can advance into higher-value roles in the company over the longer term. For example, Starbucks grows baristas into some of the industry’s best store, district and regional managers. P.F. Chang looks at the dishwasher role as a point of entry to test and grow talent that they then rotate into increasingly high-value roles in their restaurants.

While hiring for potential makes logical business sense, it can easily lead to bias if the factors associated with potential for job promotion are left undefined. Managers can, consciously or unconsciously, default to seeing potential as “someone like me,” and hire on the basis of factors irrelevant to the long-term capacity to perform and advance. If, however, potential can be pinned down to specific, measurable qualities, it becomes a powerful basis for data-driven hiring.

To take an example from our study, let’s take

a look at the matches for the customer service role. Within this sample, one can map the knacks of resourcefulness—an important foundation for day-to-day performance in this job— and logical reasoning, which we could hypothesize is an important factor in the potential to advance into supervisory roles involving greater focus on process design and operational decision-making. Figure 7 is a scatter-plot of the youth who matched the customer service role, showing where they score on these two knacks. A company focused just on ensuring good performance in the current job might select candidates based on resourcefulness (the pink shaded section of the plot). Given the larger number of individuals who fulfill these less stringent criteria, the employer could look to drive excellence on operating metrics such as low time to fill open positions. A company balancing focus on long-term potential with performance in the immediate job at hand might draw from the smaller set of candidates in the red square, who are strong both in resourcefulness and logical reasoning.

As described in the discussion of the “second myth” above, opportunity youth have a similar distribution of knacks to the general population. For any given target group of aptitudes (e.g., people person + planning + logical reasoning) we see about the same numbers who score high in our group as we’d see in the general population, and the same kinds of interrelationships among aptitudes (e.g., positive correlation among certain cognitive abilities) one would expect based on the broader research literature. When an employer commits to the path of delineating what potential really means and to measuring the relevant underlying pattern of talent markers as objectively as possible, that opens the door to finding talent beyond their traditional recruiting sources who have the aptitudes relevant for long-term advancement.

low high

low

high

Logical Reasoning

Res

ourc

eful

ness

Customer Service Job Matching

Multiplejobs

At leastone job

Male

Fema

le

Multiple jobs, avg

At least one job, avg

0%

25%

50%

75%

100%

Perc

ent o

f you

th m

atch

ing

jobs

Job−matching rates by ethnicity and gender

American Indian /Alaska Native Asian Black /

AfricanAmerican

Hispanic Native Hawaiian /Other

Pacific IslanderWhite

1817

REALITY: Quantitative approaches to hiring can be designed that are fair, and blind to differences of gender and ethnicity. In our study, youth of all genders and ethnicities demonstrated at similar rates the aptitudes associated with high performance in four distinct entry-level roles.

This debunks a sixth commonly-held myth: that an objective, quantitative approach to hiring will disadvantage certain gender or ethnic groups. Rather, the true enemy is the

bias that can easily creep into subjective hiring decisions.

There’s a large body of literature demonstrating the prevalence of bias—often unconscious—throughout the hiring process. Examples of bias exist for many factors. Women are less likely to be considered or hired into specific positions, and they receive less pay and fewer promotions than men. 5,6 Job applicants with “African-American-sounding names” have been found to receive 50% fewer responses to from initial resume submissions.7

Employers who systematize their selection process more, in some cases driven by the objective of avoiding bias and in many cases

driven mostly by the objective of minimizing the cost of recruiting, often let bias in through the back door by basing screening decisions on factors such as education or job experience. These factors don’t systematically drive job performance and do significantly differ by race/ethnicity and by socioeconomic background.

5. Uhlmann, Eric L., and Geoffrey L. Cohen. “Constructed Criteria: Redefining Merit to Justify Discrimination.” Psychological Science, Vol. 16, No. 6 (2005), pp. 474-480. Print.

6. Biernat, Monica, and Diane Kobrynowicz. “Gender- and race-based standards of competence: Lower minimum standards but higher ability standards for devalued groups.” Journal of Personality and Social Psychology, Vol. 72 (3) (1997), pp. 544-557. Print.

7. Bertrand, Marianne, and Sendhil Mullainathan. “Are Emily and Greg More Employable than Lakisha and Jamal?” American Economic Review, Vol. 94, No. 4 (2004), pp. 991-1013. Print.

Figure 8. Youth of all ethnicities and genders show comparably high rates

of positive matching for jobs

Percent of youth by gender (green: male, blue: female) and ethnicity qualifying for multiple

jobs (dark green/blue) or at least one job (light green/blue) at the basic competency level.

Bars do not reach 100% as a small proportion of youth do not match any jobs. The two lines

at the top show the average across ethnicities of youth who qualify for at least one job (99%)

and multiple jobs (89%).

Figure 7. Youth demonstrate aptitudes associated with advancement potential

Youth Logical Reasoning (horizontal axis) and Resourcefulness (vertical axis) knack scores

are shown for each individual. Pink shaded area indicates candidates an employer might

consider for near-term performance potential based on their high resourcefulness. Red

square indicates candidates who score well for near-term performance potential as well as

advancement potential based on their logical reasoning ability.

The reality is that aptitudes tend to be distributed in the same way across different demographic groups—which is precisely the reason bias is so concerning. What we see in Knack’s data is exactly what we’d expect: similar distributions of knacks by gender and by race/ethnicity.

2019

THE DAMAGE THE MYTHS DO: Systemic Failures in Talent Markets

Large numbers of youth—and, no doubt, many older adults as well—have attributes that are predictive of high performance in specific jobs that are missed by current selection processes. These breakages occur across the lifecycle of the hiring process.

• Employers often don’t rigorously identify the attributes that are most relevant to performance in entry-level roles or most differentiating in terms of potential for longer-term advancement. As a result, they rely on proxies that don’t predict performance—e.g., tenure in a similar role in another company, education level—and that impede access for youth who have job-relevant aptitudes but don’t fulfill the employer’s criteria.

• Employers often fish in the same limited set of talent pools, missing out on talent from other sources. Many of the youth in our study are poorly connected to the channels that lead to employment: they don’t have relationships with current employees that lead to referrals, they aren’t affiliated with institutions that employers recruit from (e.g., higher education) and they aren’t knowledgeable and confident navigating the job search ecosystem (e.g., using online resources, working with agencies).

• Once employers advance candidates beyond a first screen, their selection processes often fail to extract a relevant signal (e.g., direct measurement of job-relevant aptitudes) and instead introduce significant noise (e.g., subjective impressions from interviews).

These data suggest the existence of a market failure in entry-level employment.

This pattern is costly for employers, for jobseekers, and for the economy as a whole. Jobs remain unfilled for longer periods of time. New hires come onboard who predictably perform less well than others who could have been selected. Retention rates suffer because employers hire those people who already have the best outside alternatives, rather than offering opportunities to others who will value the opportunity more based on the difference the work makes to their long-term prospects.

And this pattern is even more costly at a broader social level. When youth fail to gain the right employment opportunities early on, their future employment outcomes suffer considerably. They are more likely to require expensive social assistance for longer periods of time. Their health outcomes worsen, as do the health outcomes and economic prospects of their children.8 We lock in inequality in ways that undermine our broader national ideals.

8. Bielfeld, Clive R., Henry M. Levin, Rachel Rosen, 2012. “The Economic Value of Opportunity Youth.” Washington, D.C.: The Corporation for National and Community Service and the White House Council for Community Solutions.

2221

A PRESCRIPTION: Data-Driven Hiring for Entry-Level Jobs

This requires directing attention to each stage of the recruiting process and employee lifecycle, not unlike the rigorous focus on continuous improvement that has made functions like manufacturing, supply chain management and marketing so much more productive in recent years. Specifically, this includes:

As so many fields have in recent years, entry-level hiring must make the transition from relying on human intuition to leveraging the power of scientific, data-driven processes.

• Making the best inferences from performance data of current employees back to the identification of talent requirements—which patterns of traits and abilities are most important in driving results on the job

• Identifying a wide array of talent pools from which to draw candidates, directly and/or through online and offline intermediaries

• Evaluating candidates in ways that deliver valid, unbiased signals regarding the markers most important for job performance

• Measuring performance on the job in a granular, repeatable, and systematic way – and leveraging this data to feed back into both job design and the predictive signals that drive the hiring process

There are more tools and better practices available today than ever before to identify human potential and to manage human capital. Often, the limiting factor is the institutional will for companies to make a commitment to data-driven hiring, build the relevant accountabilities into their line management and HR organizations, and shift the day-to-day basis for decision-making from intuition to data.

Individual employers have a great deal to gain from the application of talent analytics. The gains from infusing predictive talent data into talent markets are even greater when we look at the broader social consequences. Talent markets exchange our most valuable resources, but they are compromised by severe systematic failures: bias, information gaps and poor decision making stand in the way of individuals who could be strong performers. These market failures most greatly affect the disadvantaged: individuals who are disconnected, vulnerable to discrimination and lacking in the markers (e.g., experience and education) that employers traditionally value. Youth are particularly likely to be hurt by such market failures. Not only are their strengths often invisible to employers who have not put tools in place to see them, but

often these youth don’t have insight into the nature of their own true potential and what undreamt-of opportunities that potential could allow them to access.

Solving our nation’s youth employment challenge will require advancing the ways that talent markets work to identify young people with the specific aptitudes that drive success on the dimensions employers value. In order to promote innovation that works to bridge these gaps, the social sector must work closely with employers while ensuring that the resulting tools successfully reach the populations most in need. Data-driven hiring will solve one critical part of a systemic problem. A range of other supports will be required to address the specific gaps in job readiness and challenging life circumstances that distinct segments of opportunity youth face. The more that provision of these supports can also be informed by data and analysis, the better we will be able to move the ecosystem of youth employment from a cottage industry of local programs to a scalable, market-driven effort. From both the perspective of human need and the perspective of economic value, this is an opportunity from which we cannot turn away.

24

APPENDIX

Technological advances in predictive talent analytics are increasingly making the promise of data-driven talent matching and hiring a reality for employers and youth alike, rapidly changing how talent is recruited and selected across a range of employers.

These new tools can also advance social and economic equality by identifying chronically disenfranchised job seekers with high potential and matching them to well-fitting jobs based on objective and predictive signals and criteria. Such an approach can help employers sidestep various forms of bias and focus the recruiting process only on those selection criteria that are truly relevant for employee performance on the job.

To the best of our knowledge, the promise of these tools has not yet been tested in the context of youth employment. The Rockefeller Foundation and grantee Incandescent therefore partnered with Knack, an innovator in the talent analytics space, to conduct an innovative pilot study.

This study was designed to test if youth with limited educational qualifications and work experience can, when evaluated using these new tools, demonstrate traits and abilities that would make them attractive hires for entry-level jobs. Using Knack’s games to predict performance in specific entry-level roles, we tested the number of matches between the aptitudes of a sample of youth and the distinct predictors of high performance in each of a range of jobs. These matches are often missed today because employers overweigh education and experience as proxies for performance potential, along with subjective, bias-prone judgments of resumes and interviews.

While we attempted to conduct this study in a rigorous and objective manner, our work should not be considered scientifically conclusive. It is rather a demonstration of what these tools could bring to the field of youth employment. We hope that this effort leads to further exploration of ways in which these tools can help advance youth employment and data-driven recruiting.

Study ParticipantsFrom March to May 2015, Incandescent enrolled 605 participants at 18 workforce development centers to take part in the Knack pilot. Participants were young adults currently enrolled or recently graduated from a workforce development program in their local area.

• 405 Participants at 14 centers were from the City of New York’s Young Adult Internship Program (YAIP), funded by the Department of Youth and Community Development (DYCD)

• Participating Centers: The Door, LaGuardia Community College, Northern Manhattan Improvement Corporation, Henry Street Settlement, Cypress Hills Local Development Corporation, Catholic Charities of Brooklyn & Queens, Opportunities for a Better Tomorrow (Bed-Stuy, YRoads and Bushwick), Chinese-American Planning Council, FEGS, Mosholu Montefiore Community Center, Supportive Children’s Advocacy Network and BronxWorks

• 66 Participants at three centers were from New York in programs not affiliated with the DYCD YAIP program

• Participating Centers: Per Scholas, Comprehensive Development Inc., Grace Institute

• 134 Participants at one center were from the Los Angeles Job Corps Center in the Los Angeles metro area

• All participants were between the ages of 18-24 and were not currently employed or enrolled in full-time education. The majority of participants (78%) had a high school diploma. Program partners were asked to provide a representative sample of the youth in their programs. The overall sample of 605 participants had the following characteristics:

• Gender: 51% male, 49% female

• Race / Ethnicity: 46% African American, 37% Hispanic, 7% White, 10% Other

Study StructureBetween 10-50 youth participated simultaneously in each session, which took between 60-90 minutes to complete and consisted of an introduction, gameplay and a survey:

• Introduction (10 minutes): The researcher gave a brief introduction to The Rockefeller U.S Youth Unemployment initiative and provided specific information about Knack games. Participants then had the opportunity to ask any questions prior to reading through and signing their consent forms.

• Tablet-Based Data Collection (35-50 minutes): Each participant played three Knack games – Balloon Brigade (BB), Wasabi Waiter (WW) and Meta Maze (MM). They were instructed to complete BB and WW and to play MM for 15-20 minutes (depending on how quickly they completed the other games).

• Survey (15 minutes): The survey included basic demographic questions (e.g., age,

race/ethnicity) as well as questions about their employment and education history.

Participation was completely anonymous, and each participant was given a $15 cash gift card as a thank you for their participation. At the end of the gameplay, all tablets used were donated to the participating organizations. Knack Analytic MethodologyKnack analyzed the data to provide insight on the behavioral markers and job performance potential of the participating youth. Knack’s innovative technology focuses on analyzing gameplay data in great detail. Gameplay produces a high-volume data stream that paints a powerful, high-resolution portrait of how the player explores, evaluates, reacts, decides, plans, and learns. From these data points, Knack looks for general patterns that relate to reliably distinctive behaviors. For all the youth in this study who played Knack’s games, Knack was able to build a profile of their abilities and traits, known as knacks.

In addition, we compared youth performance to Knack’s proprietary database of job-specific ultraknacks, which predict performance in a particular job at a particular company. These ultraknacks are created by analyzing the gameplay data of current employees and using performance data provided by the company (e.g., supervisor’s job performance ratings) to identify behavioral patterns in the data that distinguish top performers. For example, current customer service representatives for a global financial services company played Balloon Brigade in fall 2014.

Knack analyzed gameplay data from a sample of employees to build a predictive model of job performance and then tested this model against the rest of the employee sample to evaluate its explanatory power to predict performance. This process uses machine learning techniques to identify the behavioral

23

2625

patterns of high performers that separate them from low performers, while ignoring patterns that high and low performers have in common or that only single out individuals.

Ultraknacks are designed to help companies identify prospective employees with similar high-performance potential in populations beyond current job holders. Thus, the models need to be generalizable and to provide unbiased predictions of potential. Knack uses a rigorous vetting process for all models, testing and confirming accuracy in large and diverse groups of people to ensure there is no advantage to experienced gamers or individuals based on gender, ethnicity, or age. Knack uses multiple methods to ensure this outcome:

• Knack’s games are designed for a casual user: even inexperienced players can learn how to play the games in seconds

• Knack builds models using diverse populations, and as a result they are explicitly forcing the models to ignore behavioral signals that map to these demographic factors or gaming experience

• Knack double-checks whether bias exists in the knack results to ensure the algorithms use only behavioral signals that predict outcomes accurately for everyone

To revisit the customer service ultraknack, Knack did an extensive study of its performance in protected demographic groups based on ethnicity, age, and gender. Various tests for adverse impact, including the two most commonly cited as industry standards (80% rule and 2 standard deviation rule), were conducted for each group:

• 80% rule: Is the pass rate of a protected group less than 80% of that of the non-protected group?

• Two standard deviation rule (Fisher’s exact test): Is the protected group’s pass rate more than two standard deviations lower than that of the non-protected group?

• Two-sample difference in means (2-sample T-test): Is the mean ultraknack score of a protected group significantly less than the mean score of the non-protected group?

All of these tests revealed no underlying bias or adverse impact on protected groups for the customer service ultraknack model.

For the youth in this study, Knack used four ultraknack models that had already been created:

• Customer Service: Customer-facing service representatives at a global financial services company

• Claims Processor: Back-office functions for insurance operations at a global financial services company

• Restaurant Servicer: Customer-facing role in a California restaurant chain operator

• Financial Analyst: Higher-skill job requiring financial data analysis for a global insurance company

Youth were given a score from 0 to 100 on each of the four ultraknack models, where the higher the score the greater the potential to be a high performer in each of these roles. We then compared the youth scores to the employee populations used to create the ultraknacks, allowing us to determine how many youth scored above the average score of the entire employee population or above the average score of the high-performing employee population. These two score markers are often used by Knack’s customers when evaluating ultraknack scores during the hiring process, so they indicate how a potential employer might view Knack data for the youth in the study.

Figure 9. Screenshots of the three Knack games used in this study

The authors would like to extend special thanks to the study participants, as well as to the following individuals who have, through their advice, support and active effort, helped make this paper and the study it is based on possible:

The Rockefeller FoundationEvan O’Donnell Alyson Wise

IncandescentCharlotte DobbsJacob JuanEric Liu Darko LovricMark McCoyDiana Spasenkova

KnackChris Carroll Michelle Earle Sherri FlippoJohn Funge Thomas Jerde Gretchen McDougallJennifer Hasenyager Smith Steve Woyach

Margaret Aylward, Comprehensive Development, Inc.

Andrea Vaghy Benyola, The Door

Koji Chavez, Stanford University

Victoria Franklin, National Job Corps Association

Adjoa Gzifa, LaGuardia Community College

Jackie Honore, Los Angeles Job Corps Center

Farhana Hossain, MDRC

Thea Kennedy, Design Quixotic

Andreé Lockwood, Comprehensive Development Inc.

Jane MacKillop, LaGuardia Community College

John Mancuso, Comprehensive Development Inc.

Melissa Mar, Melissa Mar Design

Louis Miceli, JobsFirst NYC

Marjorie Parker, JobsFirst NYC

M’Shell Patterson, LaGuardia Community College

New York City Department of Youth and Community Development

Kelly Richardson, Per Scholas

Reshard Riggins, The Door

Sid Ross, Grace Institute

Anand Vimalassery, Ellipsis LLC

Takiyah Weekes, Henry Street Settlement

Evan Wittenberg, Box

ACKNOWLEDGEMENTS