white paper_step+ for bpo industry_dec2014

Recruiting and Hiring with the Standard Test for English Professionals Plus (STEP+):

Research for the Business Process Outsourcing (BPO) industry

Business Solutions

December 2014

Business Solutions Recruiting and Hiring with the Standard Test for English Professionals Plus (STEP+)

2

Contents

3 Executive Summary 4 Background5 Validity Research5 Study 1. STEP+ and the Versant English Test6 Study 2. STEP+ and the Versant Writing Test7 Reliability Research7 Study 3. STEP+ and the One Placement Test10 Efficacy Research10 Study 4. Differences in STEP+ scores by job performance (India)12 Study 5. Differences in STEP+ scores by job performance (Australia)13 Study 6. STEP+ scores as predictor of training and job performance16 Recommendations


3

Pearson English Business Solutions conducted six studies with a large, multinational business process outsourcing (BPO) organization that uses the Standard Test for English Professionals Plus (STEP+) in key markets such as India, the Philippines, and Latin America. Taken together, these studies provide rigorous and diverse evidence that STEP+ is a reliable and valid test of English proficiency that can be used effectively as part of a BPO company’s hiring process.

Executive Summary

– Study 1 found a moderate positive correlation between STEP+ and the Versant English Test, which is empirically validated and widely used in the BPO industry.

– Study 2 found a limited relationship between STEP+ and the Versant Writing Test. This is expected, because the two tests concentrate on different English skills. The result supports the discriminant validity of STEP+.

– Study 3 compared STEP+ with a highly similar test, the One Placement Test. The two assessments had a significant positive relationship, and the observed pattern of lower scores on the One Placement Test was hypothesized to occur due to regression to the mean (a statistical phenomenon), test-taker preparation for STEP+, and measurement error.

– Studies 4 and 5 investigated the relationship of STEP+ with job performance by examining high- and low-performing customer service agents in India and Australia. High performers achieved significantly higher scores on STEP+ than low performers. The results support the efficacy of STEP+ as a hiring tool.

– Study 6 explored the efficacy of STEP+ as a hiring tool by comparing new hires’ STEP+ scores with their training and on-the-job performance scores. STEP+ scores were positively correlated with agents’ performance in general English communication training, and with customer satisfaction during their first three months on the job.

Lindsay Oishi, PhD

Product Owner - Efficacy and Engagement

Pearson English Corporation


4

Business process outsourcing (BPO) companies often use English language assessments to screen applicants for customer service jobs. For BPO organizations, hiring Customer Service Representatives (CSRs or “agents”) with proficiency in English is a core component of profitability. Employing agents with strong English skills is a service differentiator that drives sales, a contractual requirement for many BPO clients, and a significant factor in costs and operating income.

An effective assessment solution supports several important business outcomes for BPO organizations. First, it limits the costs associated with hiring unsuitable candidates, re-testing, and training candidates who need extensive remediation in English. Second, high-quality English assessments can help to identify and retain valuable applicants in highly competitive labor markets, such as India and the Philippines. Third, an appropriate assessment of English proficiency can be used to set and monitor a baseline level of the language skills required for effective customer service and support.

Pearson Business English Solutions offers the Standard Test for English Professionals Plus (STEP+) as a hiring tool for BPO and other organizations. STEP+ is web-based, 52-minute test of Business English that includes sections on grammar, listening, reading and speaking. The test is based on Canale and Swain’s model of communicative competence1, and presents items in global business contexts, such as meeting with colleagues and making telephone calls. STEP+ reliably and validly measures an individual’s ability to understand and use English for business purposes2, but has not been evaluated for efficacy with regard to hiring customer service representatives.

To understand efficacy in this context, we collaborated with a large, multinational BPO client to conduct six studies, from 2012 to 2014, which evaluated different aspects of the STEP+ implementation. Studies 1 through 3 examined assessment validity as measured by correlation with other English tests: the Versant English Test, the Versant Writing Test, and the Pearson One Placement Test. Studies 4 through 6 looked at the relationships between STEP+ scores and agent outcomes during initial training and the first year of employment. The six studies spanned multiple geographic regions, including India, the Philippines, Latin America, and Australia. The results indicated that STEP+ can be used effectively as part of BPO hiring processes, while indicating several ways to enhance efficacy through product implementation.

Background

1 Canale, M., and Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1, 1–47. 2 Oishi, L., and Liao, S. (2013). The GlobalEnglish Standard Test of English for Professionals Plus (STEP+): Development and Validation. Available online at http://static.globalenglish.com/files/reports/STEP_Plus_Whitepaper.pdf.


5

3 Pearson (2011). Versant English Test: Test Description and Validation Summary. Available online at http://www.versanttest.com/technology/VersantEnglishTestValidation.pdf.

Study 1. STEP+ and the Versant English Test

Prior to implementing STEP+, the BPO had used the Versant English Test (VET) for pre-hire language assessment. The VET takes 15 minutes and provides automated assessment of general English speaking and listening skills. To explore how STEP+ results compared to the VET, the BPO had 20 agent candidates in Latin America take both STEP+ and the VET in June 2013. The correlation between the two tests’ total scores was .51, p < .05 (Figure 1). Furthermore, 19 of the 20 candidates responded on a post-test survey that STEP+ results were an accurate representation of their current English proficiency. This small study provided initial evidence that STEP+ is likely to be as effective for BPO hiring needs as the VET, a market-leading assessment with extensive empirical validation.3

Figure 1:Relationship of STEP+ and Versant English Test (N = 20)

STEP+ Range: 0 - 1,581Versant English Test Range: 20-80Adjusted r-squared = 0.22, p < .05.

Validity Research


6

Study 2. STEP+ and the Versant Writing Test

Candidates for a text-based chat support program in the Philippines (N = 39) took both STEP+ and the Versant Writing Test (VWT) in separate testing sessions during May and June 2014. The VWT is an online, 35-minute assessment that focuses on general English written production, with some recruitment of reading and listening skills; it does not test speaking. As shown in Figure 2, the correlation between total scores on both tests was .29, p = .072. The relatively low correlation is expected, because the tests do not measure the same English language skills. These results suggest that STEP+ is less appropriate than the Versant Writing Test for hiring agents whose primary job function involves written production.

Figure 2. Relationship of STEP+ and Versant Writing (N = 39)

STEP+ Range: 0 - 1,581Versant Writing Test Range: 20-80Adjusted r-squared = 0.06, p < .08.


7

Study 3. STEP+ and the One Placement Test

Candidates (N = 952) took STEP+ from June through December 2013 as part of the hiring process for three different customer service programs in India. All candidates had to meet a minimum score requirement on STEP+ (Level 7, raw score of 917). Those who met this requirement then took the One Placement Test at the start of the training session, which typically began within one to two weeks of hiring. The One Placement Test is based on the same test construct as STEP+, and has highly similar items in three of the same sections (Grammar, Listening and Reading); however, the One Placement Test does not have a Speaking section. There was a moderate positive correlation of .23 between the total scores for both tests (p < .001), as shown in Figure 3 and detailed in Table 1. These results support the test-retest reliability of STEP+.

Figure 3. Relationship of STEP+ and One Placement Test (N = 952)

STEP+ Score Range: 0 - 1,581One Placement Test Range: 0 - 1,190

Adjusted r-squared = 0.05, p < .001.

Reliability Research

STEP+

Total Grammar Listening Reading Speaking

One Placement Test

Total .23** .11* .14** .05 .19**

Grammar .14** .06 .05 .16**

Listening .11* .02 .15**

Readingl .06 .14**

Table 1. Correlations between STEP+ and One Placement Test (N = 952)

*p < .01, **p < .001


8

In many cases, however, individuals obtained higher scores on STEP+ than they did on the One Placement Test. On average, the STEP+ subtotal (without the Speaking section) was 177 points higher than the One Placement Test total score. This difference in means was significant, t (951) = 32.31, p < .001. Higher STEP+ scores were were also observed in the Grammar and Reading sections. Table 2 shows the means, standard deviations and paired t-test results for STEP+ and Placement Test scores.

Mean (SD) Paired t-test

STEP+^ Placement Test t value p

Total 860 (91) 683 (158) 32.31 < .001

Grammar 320 (43) 234 (61) 38.07 < .001

Listening 273 (55) 267 (68) 2.36 Not significant

Readingl 267 (62) 182 (74) 27.88 < .001

Table 2. Descriptive statistics and t-test results for One Placement Test and STEP+

^ Excluding the Speaking sub-score so that the possible score range is equal (0 – 1,190 total score)


9

4 Barnett, A. G., van der Pols, J. C., & Dobson, A. J. (2005). Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology, 34(1), 215-220.

There are several possible explanations for the observed differences in One Placement Test and STEP+ scores. The first is regression to the mean, which, according to Barnett, van der Pols and Dobson, “is a statistical phenomenon that occurs when repeated measurements are made on the same subject or unit … It happens when unusually large or small measurements tend to be followed by measurements that are closer to the mean.”4

In this case, all individuals were selected to have a high STEP+ score, meeting the minimum threshold of 917 (Level 7). It is likely that a significant portion of candidates were testing at the maximum of their ability or even beyond. Test results that indicate that an individual has higher proficiency than his or her true proficiency can be a result of luck, cheating, or intensive test-specific preparation. Re-testing only those individuals who score on the high end of a test will consistently reveal that some people have lower proficiency than previously indicated.

As the largest difference between One Placement Test and STEP+ performance was observed in the Grammar section, it is possible that candidates were receiving information about that section’s items for STEP+ in advance. The existence of “cheat sheets” and other preparation aids for high-stakes tests in India is well known. According to recruiters and trainers working in India, there is also a robust word-of-mouth system by which other recruiters and employment agencies spread information in order to maximize the chances of their candidates getting hired.

A second possible explanation for the observed differences between the two tests is test-taker effort. When taking STEP+, individuals were applying for a competitive job and knew that failing to meet the required score would result in their applications being rejected. In contrast, when taking the One Placement Test, agents had already started training for the job, and had taken a very similar test within the last few weeks. Test fatigue and the lower-stakes context may have limited their performance on the One Placement Test.


10

Low Performance High Performance

Group size (N) 56 62

Average job performance* 67% 120%

STEP+ Mean (SD) 890 (179) 978 (172)

STEP+ Level (SD) 6.3 (1.25) 6.9 (1.19)

Table 3. Descriptive variables for low and high performance groups

* Job performance was defined as percentage of target met across four variables.

5 Variables included average time to handle a service call, first call problem resolution percentage, and two measures of customer satisfaction.

Efficacy Research

Study 4. Differences in STEP+ scores by job performance (India)

To investigate the efficacy of STEP+ as a predictor of job performance for new customer service representatives, we worked with the BPO on a score-benchmarking project in India. We tested 118 agents at two different job performance levels, as measured by standard call center industry variables.5 High performers exceeded job requirements, while low performers failed to meet targets. All participants had been working in the same voice-based support program for 60 to 180 days. Table 3 shows that on average, the high performers scored 88 more points on STEP+ than the low performers.


11

We analyzed the relationship between STEP+ scores and both job performance and tenure on the job. Agents were divided into two groups: 60 to 90 days on the job (low tenure), or 91 to 180 days on the job (high tenure). A 2 x 2 analysis of variance (ANOVA) of job performance and tenure on STEP+ total scores resulted in a significant effect of performance alone, F (1, 111) = 2.23, p < .01. Overall, the high performance group did better on STEP+ than the low performance group, regardless of their time on the job. This result supports the use of STEP+ as an informative measure of the English skills needed for success as a Customer Service Representative.

As shown in Figure 4, the mean STEP+ total score among low performers with more than 90 days on the job was higher than the mean for low performers with only 60-90 days on the job. It is possible working as a customer service representative may have a positive effect on English proficiency; however, this will have to be investigated in future studies, as the effect did not reach statistical significance.

Figure 4. Relationship of job performance, tenure and STEP+ (India). Bars represent standard errors.


12

Study 5. Differences in STEP+ scores by job performance (Australia)

We repeated the high/low performer analysis of STEP+ scores in Australia during December 2012 and January 2013. A sample of 77 agents who exceeded performance targets was compared with a sample of 72 agents who ranked at the bottom of their programs. For these programs, job tenure was recorded as low (less than 120 days) or high (121 days or more).

A 2 x 2 ANOVA (performance x tenure) on STEP+ total scores resulted in a significant effect of performance alone, F (1, 143) = 6.13, p < .05. The STEP+ total scores for the high performance group (M = 1034, SD = 171) were higher on average than the STEP+ scores for the low performance group (M = 968, SD = 153). There were no differences by tenure for either performance group, as shown in Figure 5.

Figure 5. Relationship of job performance, tenure and STEP+ (Australia)Bars represent standard errors.


13

Study 6. STEP+ scores as predictor of training and job performance

Working with the BPO organization in the Philippines, we examined the efficacy of STEP+ as a predictor of agent success in two contexts: job training and on-the-job. For this study, agents were applying or working for a text-based chat support program in the second half of 2013. Agents were evaluated during two stages of training. The first training period covered basic communication and customer service in English topics. The score for the first training period was an average percentage correct across a number of topical English quizzes and tests. The second training period developed candidates’ knowledge and skills in a specific customer service program (e.g., technical support for an Internet company) and was evaluated through a written final assessment. Table 4 shows that there was a moderate positive relationship between STEP+ and candidate performance in both the first and second training programs.

NAverage Score

(Range)

Correlation with STEP+ Total

Scorep-value

r-squared (Adj.)

Training program 1: Communication and customer service in English

32584%

(42 - 99%).24 <.01 .07

Training program 2: Program-specific topics and skills

3090%

(76 - 100%).38 <.05 .12

Table 4. Relationship between STEP+ total score and training performance


14

Next, STEP+ scores for agents who passed training were compared with their job performance. Eighty agents had one to three months of performance data between October and December 2013. The job performance variables were a measure of customer satisfaction (willingness to recommend) and a measure of effectiveness (rate of first call resolution). There was an observed trend towards a positive relationship only between STEP+ score and customer satisfaction, r (80) = .21, p = .07. Figure 6 illustrates this relationship by showing the average customer satisfaction rating was higher for agents who scored within Levels 8 and 9 on STEP+ (raw score of 1065-1399) than for those who scored at Level 7 (raw score of 917-1064).

Figure 6. Average customer satisfaction by STEP+ English levelBars represent standard errors.


15

As tenure may affect job performance, we performed a 2 x 2 ANOVA (level x tenure) to determine the relative contribution of each variable to average customer satisfaction (willingness to recommend). The level conditions were Level 7 (n = 58) or Levels 8 and 9 (n = 22), and the tenure conditions were Low (3 months or less) or High (3 to 8 months). The results indicated a trend towards an effect of level, F(1,76) = 3.76, p = .06, and a significant effect of tenure, F(1, 76) = 14.85, p < .001. There was also an interaction between level and tenure, F(1, 76) = 4.76, p < .05. As shown in Figure 7, the data suggest that agents with higher English language proficiency were more successful initially, but that over time, the difference lessened. In essence, agents hired at Level 7 may “catch up” with agents hired at Levels 8 and 9, most likely due to improving English on the job.

Figure 6. Average customer satisfaction by STEP+ English levelBars represent standard errors.


16

Recommendations

Best use of STEP+ and Versant language assessments STEP+ is appropriate for assessing customer service candidates who will work in programs that require comprehensive English reading, listening and speaking skills. The Versant English Test (VET) is a viable alternative for situations that require shorter testing times and can tolerate less comprehensive test results, focused primarily on speaking. For programs that require understanding and producing written text (i.e., text-based chat support), the Versant Writing Test (VWT) is more appropriate.

Best use of STEP+ and the One Placement Test During the hiring process, STEP+ is preferred for its Speaking section, which the One Placement Test does not include. The One Placement Test can be used to identify training needs, but is only recommended when STEP+ results are significantly compromised, due to suspected measurement error such as cheating or technological failure.

Test-retest score differences In any situation when candidates who initially met a score cutoff on STEP+ are re-tested with STEP, STEP+ or the One Placement Test, it is likely that significant regression to the mean (lower scores) will be observed. Re-tests should not be administered unless there is a strong reason to believe that the first score was not accurate.

Getting the most accurate scores from STEP+ Due to the high frequency of measurement error in applied settings, such as third-party recruiting centers in India, BPO organizations will get the best results by supplementing STEP+ results with live interviews. When possible, meeting a minimum score on STEP+ should be used as a pre-requisite for a human interview. STEP+ should not be used as a final requirement after all other requirements have been passed; this reduces its usefulness and may be redundant.


17

Reducing cheating and other forms of measurement error on STEP+ As a result of this analysis, Pearson recommended that the BPO partner organization implement stronger anti-fraud measures, including:

– Using modern browser technology (Internet Explorer 8 or higher), which enables key test protections. These include detecting any user attempt to navigate away from the test interface, and requiring the test-taker to complete the assessment without pausing.

– Training recruiters to perform random test checks procedures and implementing virtual monitoring. While conventional test proctoring is impractical in the BPO hiring context, random in-person checks and virtual monitoring can be used to support overall quality assurance and discourage cheating.

– Training interviewers to probe test results, identify potential indicators of cheating, and explain unusual or inconsistent scores.

– Monitoring aggregate STEP+ performance among candidates hired through third-party recruiting agencies against BPO-managed hiring centers. It is critical to monitor STEP+ results on various dimensions (e.g., location and recruiter) to identify trends in measurement error. For example, some third-party agencies may have poor anti-cheating and proctoring procedures, less reliable technology, or a different candidate pool. Understanding these differences in a data-driven way will allow BPOs to interpret and use STEP+ results from third-party agencies appropriately.

Expanding the labor pool with flexible STEP+ score requirements In these studies, the BPO used an already-established score threshold (Level 7) to hire agents for customer service programs. Examining the performance of agents with longer tenure, however, suggested the possibility that agents who score near the minimum threshold may catch up with agents who initially scored higher on STEP+, due to practicing English and customer service on-the-job. As hiring needs increase and the labor market becomes more competitive, BPOs may consider hiring at lower thresholds and using longer, more comprehensive training and transitioning programs to bring agents with lower English proficiency up to the level needed for high customer satisfaction and agent effectiveness.

Understanding what makes a successful customer service representative While the studies presented in this report show that English language assessments such as STEP+ are a critical part of the hiring process, it is also clear that many factors beyond English contribute to agent success. Formal assessment is necessary to provide objective, consistent and valid measurement of language skills, but is not sufficient as a stand-alone hiring tool. BPO employers would benefit from tracking other characteristics, such as educational history, empathy, problem-solving ability, and domain-specific knowledge, which, together with English assessment results, may provide a hiring profile for the most successful customer service representatives.

Business Solutions Recruiting and Hiring with the Standard Test for English Professional Plus (STEP+)

About the Author About Pearson English Business Solutions*Pearson English Business Solutions offers solutions to improve enterprise-level Business English, enable effective communication and collaboration, and drive high performance in a global economy. We offer a complete suite of formal, social, and blended learning tools to form a comprehensive and customizable learning program, as well as assessment and performance support applications. Focused on the global enterprise market, Pearson English Business Solutions supplies its services around the globe through its SaaS platform, with dedicated corporate sales and support offices in more than 20 countries. More than 500 of the world’s largest global companies are already clients of Pearson English Business Solutions in 175 countries. Pearson English Business Solutions is owned by Pearson, the world’s leading learning company. For more information visit www.english.com/business.

Dr. Lindsay Oishi is the Product Owner - Efficacy and Engagement at Pearson English Corporation. She has a BA from Georgetown University, an MA from Oxford University, and a PhD in Educational Psychology from Stanford University. Dr. Oishi has worked on research, assessment and evaluation in both academic settings, at the Hasso Plattner Institute of Design and the Stanford University Online High School, and in business settings, at Hewlett-Packard and Adobe Systems. She has written about online education and language learning in publications for various organizations, including the International Association for K-12 Online Learning, the California Foreign Language Project, and the American Educational Research Association.

* Please be advised that GlobalEnglish has changed its name to Pearson English Business Solutions.

© 2014 Pearson English, a division of Pearson plc. All rights reserved.

white paper_step+ for bpo industry_dec2014

Documents