mystery shopping pilot evaluation study - digital education

Connexions Direct:Mystery Shopping Pilot Evaluation Study

Emma StarlingTNS Mystery Shopping

Research Report RR561

RESEARCH

Research Report No 561

Connexions Direct: Mystery Shopping Pilot Evaluation Study

Emma Starling TNS Mystery Shopping

The views expressed in this report are the author’s and do not necessarily reflect those of the Department for Education and Skills. © TNS 2004 ISBN 1 84478 287 5

Connexions Direct Mystery Shopping 2004

TNS © 2004 1

Acknowledgements This pilot mystery shopping programme was carried out by TNS Mystery Shopping. We would like to take this opportunity to thank the following people who contributed towards the study: At Local Connexions Partnerships: John Bottomley, Young Peoples’ Involvement Manager, Essex, Southend & Thurrock Connexions Partnership Linda Mottram, Cheshire & Warrington Connexions Partnership Personal Advisers and young people from Essex, Southend & Thurrock Connexions Partnership Personal Advisers and young people from Cheshire & Warrington Connexions Partnership Personal Advisers from Derbyshire Connexions Partnership At Newcastle CXD: Connexions Direct Advisor staff and supervisors who contributed to the design of assessment criteria. At DfES: David Betteley, Evaluation Project Manager Lynne Ford, Connexions Direct Policy Team Michelle Turner, Connexions Direct Policy Team At TNS: The team of mystery shoppers and all of the staff who supported and monitored their work. Glossary The following terms have been used throughout this report: “CXD” Connexions Direct “The Service” Connexions Direct


TNS © 2004 2

CONTENTS

1. Executive Summary .......................................................................................................................................... 4 1.1 Introduction 4

1.2 Methodology 4

1.3 Results 5

1.3.1 By Contact Type 5

1.3.2 By Scenario 6

1.3.3 Results by Question 7

1.4 Recommendations 8

1.4.1 Mystery shoppers 8

1.4.2 Scenarios 8

1.4.3 Assessment Questionnaires 9

1.4.4 Reporting 9

2. Introduction ........................................................................................................................................... 10 2.1 Aim of Connexions Direct 10

2.2. Aim of the Pilot Mystery Shopping Study 10

3. Methodology.......................................................................................................................................... 11 3.1 The Mystery Shoppers 11

3.2 Sample 11

3.3 Scenarios 12

3.4 Briefing Instructions 13

3.5 Questionnaires 14

4. Post-Fieldwork Consultation ............................................................................................................... 16 4.1 Overview 16

4.2 Questionnaires 16

4.3 Responses 17

5. Overview of Results.............................................................................................................................. 18

6. Results by Contact Type ...................................................................................................................... 19 6.1 Telephone 19

6.2 Email 20

6.3 Webchat 21

6.4 Text 22

7. Results by Scenario.............................................................................................................................. 23 7.1 Scenario 1 - I’m thinking about going into nursing after I’ve finished school. What do I need to get in my GCSEs? 23

7.2 Scenario 2 - I’m taking my exams soon, but find it difficult to revise. Can you help me? 23

7.3 Scenario 3 - I’ve been asked to choose my options, I just don’t know what to take. 23

7.4 Scenario 4 - I left school last year, but I don’t know what I want to do now. 24

7.5 Scenario 5 - I’m worried about my mum who is drinking a lot. 24

7.6 Scenario 6 - My friend is smoking a lot of cannabis. 24

7.7 Scenario 7 - My periods are heavy and painful. Is this normal? 24

7.8 Scenario 8 - I had sex with my boyfriend and now I think I might be pregnant. 24

8. Results by Question ............................................................................................................................. 25


TNS © 2004 3

8.1 Greeting 25

8.2 Dealing With Your Enquiry 25

8.3 Overall assessment of contacts 26

9. Recommendations................................................................................................................................ 27 9.1 Mystery shoppers 27

9.2 Scenarios 28

9.3 Assessment Questionnaires 28

9.4 Reporting 28

10. APPENDIX A –Briefing Notes............................................................................................................... 30

11. APPENDIX B – Questionnaires ............................................................................................................ 36

12. APPENDIX C – About TNS Mystery Shopping.................................................................................... 40


TNS © 2004 4

1. Executive Summary

1.1 Introduction Connexions Direct (CXD) has been developed as part of the Connexions Service. It aims to provide confidential information, advice and support to 13-19 year olds on a wide range of issues, and was first launched, as a pilot service, in September 2001. Advice is accessed by telephone, webchat, email and SMS text messaging. Drawing on recommendations from an earlier evaluation of the Connexions Direct Pilot, this pilot evaluation has sought to test and appraise a method of mystery shopping of Connexions Direct to support future performance monitoring and quality assurance. 1.2 Methodology The sample of mystery shopping contacts was constructed to broadly replicate the contacts the CXD Service receives in terms of day, time and contact method. In relation to the latter, the following sample sizes were adopted: • 150 contacts were made by phone • 150 contacts were made by webchat • 100 contacts were made by email • 75 contacts were made by SMS text To ensure consistency in contact approach, the mystery shoppers were given a story or scenario framework to use. The range and content of the scenarios was agreed following consultation with Personal Advisers from the Cheshire and Warrington; Essex, Thurrock and Southend; and Derbyshire Connexions Partnerships as well as young people from Cheshire and Warrington and Essex, Thurrock and Southend. Connexions Direct advisers in Newcastle were also involved in the consultation, particularly around the design of assessment criteria (see paragraph 1.4. ‘Recommendations’ for information and discussion around the choice of mystery shoppers). Eight scenarios were used during this pilot, with the opening lines for each scenario as follows: Scenario 1 - I’m thinking about going into nursing after I’ve finished school. What do I need to get in my GCSEs? Scenario 2 - I’m taking my exams soon, but find it difficult to revise. Can you help me? Scenario 3 - I’ve been asked to choose my options, I just don’t know what to take. Scenario 4 - I left school last year, but I don’t know what I want to do now. Scenario 5 - I’m worried about my mum who is drinking a lot. Scenario 6 - My friend is smoking a lot of cannabis. Scenario 7 - My periods are heavy and painful. Is this normal? Scenario 8 - I had sex with my boyfriend and now I think I might be pregnant. Mystery shoppers were briefed to ask their opening line and then respond to the questions of the adviser. Alternative ideas for wording the opening line were given but mystery shoppers were encouraged to use their own wording to ensure that the contact appeared realistic. Immediately after each mystery shop, the shopper completed a questionnaire which served as a tool to assess and give structured feedback on contacts. In completing questionnaires,


TNS © 2004 5

mystery shoppers were reminded that contacts do not always conclude with a solution to a problem or issue, but should leave the young person feeling that they are better equipped to make a decision or move forward in relation to that issue given the additional information, support and advice offered. Mystery shoppers were also asked to take into account the different forms of contact in making their assessments, particularly in relation to SMS text contacts. Our pre-fieldwork consultation suggests that the 160-character limit on SMS contacts encourages CXD Advisers to invite texters to call the service so that they can offer more detailed advice. The methodological issue for us was around reconciling on the one hand, a practical recognition of the very different character of each mode of contact (particularly text) and on the other, the need to establish a project that would allow an objective comparison across modes of contact. Here, ongoing and future market research which explores young people’s expectations about each Connexions Direct mode of contact, could usefully inform this difficult issue. Four questionnaire versions were used, one for each mode of contact. All explored the delivery and content of the advice offered by CXD and each was split into three sections, ‘Greeting’, ‘Dealing with your enquiry’ and ‘Overall’. While questions varied slightly according to the mode of contact, most were consistent across all modes. This, in turn, allowed comparison of results across modes. The content of the questionnaires was agreed in consultation with Personal Advisers and young people from the Local Connexions Partnerships identified above and importantly, with Connexions Direct staff in Newcastle. Following fieldwork, young people and Personal Advisers from Essex, Southend and Thurrock Connexions Partnership were engaged in further consultation to gain additional views on: a) the questionnaires used for assessing mystery shops (in particular to explore what these young people and Personal Advisers regarded as the absolutely essential questions required to allow an effective assessment of the Service); b) the results obtained (in terms of whether the substantive advice offered to shoppers was ‘appropriate’ in relation to individual scenarios). 1.3 Results The following sections look at the results for each mode of contact, each scenario and by individual question from the questionnaires and, incorporates feedback received from young people and Personal Advisers in Essex, Southend and Thurrock Connexions Partnership. Results as a whole were encouraging. While the results for text mystery shops are shown along side those for the other contact methods, the reader should remain sensitive to the limits of text as a contact method relative to other modes (see above). 1.3.1 By Contact Type On average, CXD staff answered mystery shop telephone calls in 3.9 seconds. Encouragingly, a clear greeting was given in 98 per cent of calls, but only 60 per cent of callers were reminded that the service was confidential. 97 per cent of advisers appeared to listen fully to the caller. The adviser’s tone put 93 per cent of callers at ease and appropriate language was used in 97 per cent of calls. Shoppers felt that a satisfactory response had been offered in 86 per cent of calls and the information given was clear and relevant in 92 per cent. Overall a mark out of 10 of 7.7 was given, the second highest mark by contact type. On average it took 8.5 hours for CXD advisers to respond to emails. All advisers appeared to have read the mystery shopper’s email and the tone of responses put the mystery


TNS © 2004 6

shopper at ease in 97 per cent of cases. A satisfactory response was gained in 97 per cent of emails while the information given was clear and relevant in all cases. The overall mark out of 10 for email contacts was 8.1, making this the highest scoring mode of contact. Contact was made at the first attempt in 67 per cent of interactive webchats and was maintained in 57 per cent of all webchats. On average it took 76.3 seconds to respond to webchats. The advisers appeared to have fully read the mystery shopper’s question in 97 per cent of webchats and asked further questions in 82 per cent. The adviser’s tone put the mystery shopper at ease in 87 per cent of webchats and appropriate language was used in 99 per cent. A satisfactory response was gained in 83 per cent and the information was clear and relevant in 97 per cent of cases. The overall mark out of 10 was 7.3, placing this mode of contact below telephone and email. The average time to answer text messages was 49.7 minutes. CXD advisers suggested that the mystery shopper should phone CXD to discuss their issue with an adviser in 52 per cent of contacts. The adviser appeared to have fully read the mystery shopper’s text in 92 per cent of instances. The tone used put the mystery shopper at ease in 71 per cent of texts and appropriate language was used in 89 per cent, both of which were significantly less than on other modes. The overall mark out of 10 for text contacts was 6.7, making this the lowest scoring form on contact. A satisfactory response was gained in 80 per cent of texts and the information was viewed as clear and relevant in 86 per cent of cases, both of which were significantly lower than the results for all other contact modes. 1.3.2 By Scenario The following section presents the most common responses to each individual scenario. Scenario 1 - I’m thinking about going into nursing after I’ve finished school. What do I need to get in my GCSEs? In 69 per cent of mystery shops, information was offered on nursing which, as might be expected, was viewed as appropriate by Essex Personal Advisers and young people. However, Essex representatives also suggested that a referral to a local Connexions Partnership would be relevant in relation to this scenario, despite it being offered by Connexions Direct Advisers in only a very small minority of shops. Scenario 2 - I’m taking my exams soon, but find it difficult to revise. Can you help me? 57 per cent of mystery shops resulted in an offer of national information sources whilst self-help options, viewed as particularly important by Essex Personal Advisers and young people, were offered in 55 per cent of cases. The mystery shopper was advised to speak to a responsible and relevant adult, usually a teacher, in 42 per cent of contacts and local information sources, which were also viewed as helpful in Essex, were offered in 20 per cent of cases. Scenario 3 - I’ve been asked to choose my options, I just don’t know what to take. The most common response was to advise the mystery shopper to speak to a responsible and relevant adult, such as a careers adviser, teacher or parent (58 per cent). Post fieldwork consultation with Essex Personal Advisers and young people suggests the appropriateness of this approach. Interestingly, Essex Personal Advisers and young people also viewed referral to a local Connexions Partnership as an important and appropriate response in relation to this scenario. Scenario 4 - I left school last year, but I don’t know what I want to do now.


TNS © 2004 7

Local information sources were offered in 68 per cent of mystery shops, whilst national information sources were offered in 34 per cent. At the same time, Essex Personal Advisers and young people argued that referral to a local Connexions Partnership would be appropriate in these circumstances; an approach not suggested by CXD advisers for this scenario. Scenario 5 - I’m worried about my mum who is drinking a lot. National information sources were offered in 53 per cent of contacts, whilst 40 per cent were advised to speak to Mum. Interestingly, Essex Personal Advisers and young people noted that speaking to Mum may be inappropriate in the circumstances given that this might inflame the situation. Again, they argued more firmly in favour of a local referral to a Connexions Partnership Personal Adviser. Scenario 6 - My friend is smoking a lot of cannabis. The most common response was to offer national information (62 per cent). Essex Personal Advisers and young people also viewed this as the most appropriate way forward. At the same time, mystery shoppers were advised to speak to the friend in 60 per cent of contacts, though interestingly, young people and Personal Advisers in Essex did recognise that there were risks attached to such a strategy. Scenario 7 - My periods are heavy and painful. Is this normal? During 71 per cent of mystery shops, the Adviser suggested that the mystery shopper should speak to a responsible and relevant adult such as parent or doctor and this corresponded closely with the views of young people and Personal Advisers in Essex. National information sources were given in 46 per cent of cases, whilst self-help options were offered in 25 per cent of contacts. Scenario 8 - I had sex with my boyfriend and now I think I might be pregnant. The most common responses were to offer local information sources, such the details of the nearest Family Planning Clinic (47 per cent), or to suggest self-help options, such as the purchase of a pregnancy testing kit (47 per cent); both of which were rated as appropriate responses in our consultation with Essex Personal Advisers and young people. 1.3.3 Results by Question A clear greeting was given in 100 per cent of webchats, 98 per cent of calls, 96 per cent of emails and significantly less on texts, just 9 per cent (the latter arguably reflecting the limitations of SMS text). The adviser was most likely to give their name on email (95 per cent) and webchats (93 per cent) and significantly less on calls and texts (75 per cent of calls and 3 per cent of texts). Mystery shoppers were significantly more likely to be asked their name during webchats (94 per cent of webchats compared with 56 per cent of telephone calls). The percentage who were asked their age and where they live was not significantly different by mode of contact; on calls, 88 per cent were asked their age whilst 87 per cent were asked their location. Corresponding figures for webchat were 92 per cent and 85 per cent respectively. Advisers asked where the mystery shoppers had heard about Connexions Direct in 60 per cent of calls and 50 per cent of webchats, a difference between the modes which is not statistically significant. The adviser appeared to listen fully to (or read fully) the mystery shopper’s question and subsequent responses in the vast majority of contacts, with no significant differences by contact mode. Further questions were asked by the adviser during 86 per cent of calls, 82 per cent of webchats and again, significantly less on text (just 9 per cent).


TNS © 2004 8

1.4 Recommendations In this section we outline a number of recommendations for future mystery shopping of the Connexions Direct Service based on this pilot exercise. 1.4.1 Mystery shoppers Prior to the start of this pilot, discussions were held over the most appropriate mystery shoppers to use. Whilst there were some arguments for using young people known to Local Connexions Partnerships, we advised against this approach. We were concerned that by using young people from outside TNS there may be issues over their capacity to complete the work using consistent assessment criteria and according to the carefully planned sample and short timetable. In addition we questioned whether the expectations of young people who were already familiar with their Local Connexions Partnerships might be different to the ‘average’ user of the Service. At the same time however, we recognised the important need to match the characteristics of shoppers to service users to ensure credibility and realism. As such we recommended for the pilot that only existing mystery shopping-trained TNS employees should be used, drawn from the youngest age group in our panel (18-21 years) and that a smaller number of younger shoppers (14-17 years) who were children of existing TNS staff should also be used. There seems no fundamental reason why this kind of hybrid approach could not work in future mystery shopping exercises. We would also recognise the potential value of developing a more sustained relationship with an individual local Connexions Partnership to explore the possibility that they provide a supplementary source of young people who might be trained in mystery shopping techniques and undertake some of the fieldwork. This would require time and resource commitment from the local Connexions Partnership, DfES and from the research contractor to ensure careful management and achievement of an objective and transparent approach; but it would offer the advantage of increasing the pool of ‘young people’ mystery shoppers, as well as provide development opportunities for the young people concerned. We would of course also recognise the crucial role of young people and indeed local Connexions Partnership Personal Advisers (together with Connexions Direct staff) in the design of the project (including scenarios and assessment criteria) and, following fieldwork, in the analysis of the appropriateness of responses received. A more structured relationship with one or two local Connexions Partnerships would be beneficial here. Elsewhere, there is some evidence to suggest that shoppers were more persistent than ‘real’ users. In particular we would speculate that many more attempts to get through on the website were made by shoppers than would be the case by the ‘average’ user. At the same time, finding mystery shoppers to complete the work over a weekend proved time consuming and highlighted that many young people work on a Saturday. The results from other/future market research projects examining the expectations of users of the Service should be drawn upon to inform the development of an ongoing mystery shopping programme to ensure that the mystery shopping reflects the behaviour of real users of the Service. 1.4.2 Scenarios The range of scenarios reflected the type of enquiries received by the Service and the higher use of the Service by females. Eight scenarios provided a good range, but also allowed for the results to be easily examined. Future work should use the same broad categories of


TNS © 2004 9

scenario to allow for tracking over time, with minor amendments to reduce the possibility of recognition. Scenarios should however be reviewed to ensure that they continue to reflect patterns of enquiry. There is some anecdotal evidence to suggest that the advisers recognised the scenarios used as ‘mystery shopping’ calls. It is difficult to gauge the extent of recognition but assuming that there was in fact some CXD adviser recognition of scenarios, future fieldwork should be spread over a longer period than the four weeks used in the pilot, to reduce the possibility of recognition. 1.4.3 Assessment Questionnaires Future shopping exercises should allow more consultation time (drawing on the more structured relationship with a local Connexions Partnership described above, as well as consultation with CXD advisers) to ensure that the questionnaires capture all relevant aspects of the CXD performance. This consultation could also be used to prioritise questions (and their response) to enable weightings to be applied to each question at the reporting stage to reflect their perceived relative importance. The variation in expectations when using different contact types should also be examined further through market research. It is likely, for example, that expectations will be different when using text as a mode of contact compared to telephone. Similarly, young people may adapt their contact method to reflect the nature of their inquiry and again, further market research might be helpful here. 1.4.4 Reporting The Department should consider the use of these and future findings to support the performance monitoring of Connexions Direct customer service. Ideally mystery shopping should be combined with other methods, perhaps drawing on the views of service users, to make an overall assessment of service quality. However future work would also need to consider a more systematic basis for assessment of the substantive content of CXD adviser information, support and advice. Feedback from future mystery shopping should also be given to CXD adviser staff on their performance to support future staff and service development, though with no requirement to highlight the practice of individual advisers.


TNS © 2004 10

2. Introduction

2.1 Aim of Connexions Direct Connexions Direct (CXD) has been developed as a complementary and integral element of the Connexions Service. The purpose of the Service is to deliver comprehensive information, advice and support to 13 – 19 year olds on a number of issues such as careers, family issues, general and sexual health, finance, housing and drugs. It offers a confidential service for young people seven days a week from 8am – 2am. Information, advice and support is accessed through a number of different routes:

• Telephone • Live Webchat • Email • SMS Text

2.2. Aim of the Pilot Mystery Shopping Study The aim of this study was to test and appraise a method of mystery shopping of Connexions Direct to support future performance monitoring and quality assurance of the national service. This report offers a detailed review of the methods used for the pilot study and in particular, reflects on how best to engage young people in the design and delivery of a mystery shopping programme of a service dedicated to meet their needs (Sections 3, 4 and 9). The report also outlines some key findings from the pilot programme (Section 5 - 8) before going on to provide recommendations for a future mystery shopping initiative (Section 9).


TNS © 2004 11

3. Methodology

This section provides detail on how the mystery shopping programme was set-up and executed for the Connexions Direct Service. It also offers a commentary on why the particular methodological approach was adopted, outlining key considerations and issues along the way. 3.1 The Mystery Shoppers Sixteen mystery shoppers worked on this project; twelve aged 18-21 and four aged 14-17. Ten mystery shoppers were female and six were male, reflecting the greater use of the service by females and the balance of shopping scenarios (see below). The four younger shoppers were all children of experienced TNS staff. All mystery shoppers attended one of two face-to-face interactive training sessions in which • mystery shoppers were introduced to the CXD service • the purpose of the mystery shopping programme was explained • the intended approach was outlined (including details of the scenarios to be adopted by

mystery shoppers and the criteria to be used to assess shops).

The work of all mystery shoppers was subject to careful monitoring and scrutiny and additional checks were put in place to review the work of the younger group of four shoppers. These included: • parental and professional supervision during shopping contacts and assessment of calls • simple comparison by fieldwork controllers, of responses from the four younger shoppers

with the responses and assessments of older shoppers. While clearly the very small numbers involved limit effective conclusions, we found no evidence of a fundamentally different approach to assessment across these two groups. We reflect further on this approach in Section 9 below. 3.2 Sample 475 contacts were made as part of the mystery shopping programme. The sample was constructed to broadly replicate the contacts the CXD Service receives in terms of day, time and contact method. In relation to the latter, the following sample sizes were adopted: • 150 contacts were made by phone • 150 contacts were made by webchat • 100 contacts were made by email • 75 contacts were made by SMS text Fieldwork was spread equally over four consecutive weeks, starting in late February 2004. The split of mystery shops by time slot was: • 6 per cent between 8am and 11am • 26 per cent between 11am and 3pm • 42 per cent between 3pm and 8pm • 26 per cent between 8pm and midnight


TNS © 2004 12

75 per cent of the contacts were made by female mystery shoppers and 25 per cent by male. Male mystery shoppers were not used in relation to the general and sexual health scenarios which referenced issues relating to females. 3.3 Scenarios To ensure consistency in approach to each enquiry, the mystery shoppers were given a story or scenario to use when contacting CXD. The scenarios were developed to reflect the type of enquiries received by CXD. On average, just over half of calls relate to careers and learning enquiries, nearly 20 per cent relate to family and personal relationship issues and 15 per cent to general and sexual health issues. It was felt that it would not be appropriate to CXD advisers or to mystery shoppers to try to replicate crisis or despair contacts. Eight scenarios were subsequently developed; four relating to learning and career planning, two relating to general and sexual health and two relating to personal and family relationships. The range and content of the scenarios was agreed following consultation with Personal Advisers from the Cheshire and Warrington; Essex, Thurrock and Southend; and Derbyshire Connexions Partnerships as well as young people from Cheshire and Warrington and Essex, Thurrock and Southend. Connexions Direct advisers in Newcastle were also involved in the consultation. Each scenario consisted of an opening line (shown in bold below) along with background information to help the mystery shopper ‘get in to character’. A scenario ‘framework’ was offered to shoppers rather than a detailed, overly prescriptive text. This identified core aspects of the scenario but gave the shopper some limited scope to develop scenarios to make them appear less planned and in turn, reduce scope for recognition. Feedback from Connexions Direct staff in Newcastle emphasised the ‘iterative’ and ‘unstructured feel’ of many contacts and we were keen to replicate these characteristics. Scenario 1 - I’m thinking about going into nursing after I’ve finished school. What do I need to get in my GCSEs? You know very little about what routes you can go down and so you just want some general pointers. You have heard somewhere (not sure where) that you can get a degree in nursing. You are 16 and in the final year of your GCSEs. You’re hoping to get 8 GCSEs (grades A-C). You’re planning to do A-levels at school. You hadn’t thought seriously about any studying beyond A-levels. You like science at school and plan to do A-level biology. You haven’t decided which other A-levels to do. Scenario 2 - I’m taking my exams soon, but find it difficult to revise. Can you help me? You are 14 years old and are studying for your SATS that start in May (subjects include: English, Maths, Mental Arithmetic, Science). You’re worried that your friends seem able to revise, but you can’t. You are trying to revise most nights, but never seem to be able to remember what you’ve revised. You revise sitting on your bed. The house is quite noisy and you can hear your family watching TV downstairs. You’ve got a younger brother (13) who makes lots of noise in the house. Scenario 3 - I’ve been asked to choose my options, I just don’t know what to take You are 14 (year 9 of school). You like art, but your Mum thinks it’s not a good subject to study. Your mum wants you to do French/Geography/History (pick one), but you hate it. Scenario 4 - I left school last year, but I don’t know what I want to do now. You left school after your GCSEs. You’ve got 3 GCSEs (English language C, geography C and art B). You’re 17. You don’t want to do any more studying, as you didn’t like school.


TNS © 2004 13

Since leaving school you’ve been helping out your Dad with admin. at his office, but it’s boring. Scenario 5 - I’m worried about my mum who is drinking a lot. You are 15 and live with your mum, Nan and younger brother (9). Mum is getting drunk about 4 times a week. She’s drunk rather than tipsy and you and Nan have had to put her to bed. She always comes home, but this/last weekend she stayed out all night. Your Mum usually looks after you when she is not drinking. She only gets angry/moody with you when you fight with your brother, but is never violent or abusive. Your Nan does a lot around the house and is always around to cook your tea. You’ve talked to Nan, but she says mum is just upset. You haven’t spoken to anyone else. You want to be able to help her stop drinking. Scenario 6 - My friend is smoking a lot of cannabis. You and your friend are 17 and in full time education. You both started at a new college in September, although you are doing different courses, and you have a lot of new friends. Your friend is from school and you think he/she has started smoking cannabis to fit in with college friends. You are worried as your friend is an asthmatic. He/She keeps saying that it’s safer to smoke weed than drink. Scenario 7 - My periods are heavy and painful. Is this normal? You are 14 and live at home with mum and dad. You have a good relationship with dad but constantly row with mum. You’ve been to the doctors, but the doctor told you to take ibuprofen when you get the pain. You felt that they did not take you seriously because of your age. Your periods started two years ago. You would like to know if the pain will last for ever. The last one lasted for 10 days and you had to take a couple of days off school because you felt so ill. Your back hurts a lot. Scenario 8 – I had sex with my boyfriend and now I think I might be pregnant. You are 17 and had sex for the first time with your boyfriend about 3 weeks ago. You usually have fairly regular periods and are about 4 days late. You cannot remember being more than a couple of days late before. You haven’t taken a test. You don’t want to talk to your parents about it, nor would you be happy going to your local doctor in case your Mum found out. You’ve mentioned it to your boyfriend and he is scared too. 3.4 Briefing Instructions Mystery shoppers were briefed to ask their opening line and then respond to the questions of the adviser. Alternative ideas for wording the opening line were given but mystery shoppers were encouraged to use their own wording to ensure that the call appeared realistic. The mystery shoppers gave no further information about their issue beyond their opening line and were instructed not to volunteer any further information at the beginning of the enquiry. The briefing notes also contained some possible follow up questions that the adviser may ask in relation to the scenario introduced by the shopper. These were developed through general consultation with Connexions Direct Advisers in Newcastle and more directly, through discussions with the local Connexions Partnerships referenced above. Mystery shoppers were reminded that these provided a guide only and it was likely that further/different questions would be asked by the adviser. In addition they were instructed not to read the briefing information whilst talking to the adviser. With the exception of SMS text calls, all contacts were recorded/printed to support post-fieldwork analysis. A copy of the mystery shopper briefing notes can be found in Appendix A.


TNS © 2004 14

3.5 Questionnaires Immediately after each mystery shop, the shopper completed a questionnaire that served as a tool to assess and give structured feedback on contacts. The use of questionnaires allowed a consistency of approach to feedback and enabled comparisons between mystery shops to be made. The content of the questionnaires was agreed in consultation with Personal Advisers and young people from the Local Connexions Partnerships identified above and importantly, with Connexions Direct staff in Newcastle. Four questionnaires were developed, one for each mode of contact. All questionnaires explored the delivery and content of the advice offered by CXD through a combination of questions requiring a ‘categorical’ response (for example: ‘Did the CXD Adviser give their name? (Yes/No)’ and a perceptual assessment (for example: ‘Did the adviser’s tone put you at ease?’). Each was split into three sections, ‘Greeting’, ‘Dealing with your enquiry’ and ‘Overall’. Although each questionnaire had specific questions relating to the mode of contact, most questions were common to all four types. This, in turn, allowed comparison of results across modes. Common questions (featured on at least two questionnaires): Greeting How quickly was the call answered? Did the response include a clear greeting? Did the adviser explain that Connexions Direct is a confidential service? Did the adviser give their name? Dealing With Your Enquiry Did the adviser ask your name? Did the adviser ask your age? Were you asked where you live? Did the adviser ask you where you had heard about Connexions Direct? Did the adviser listen fully to your question and subsequent responses? Did the adviser ask you further questions about your query? Did the adviser offer any local information or information sources, e.g. local Connexions Partnerships? Did the adviser refer you to another service e.g. local Connexions Partnerships or other local service? Overall Did the adviser’s tone put you at ease? Did the adviser use appropriate language, whilst remaining professional? Were you given a satisfactory response to your question? Was the information clear and relevant? Did the adviser check that your question had been answered to your satisfaction? Did the adviser remind you that you can always call again if you need further help? Taking the whole call into account, what mark out of 10 would you give this call?


TNS © 2004 15

Specific questions (featured on one questionnaire only; to reflect the particularities of that mode of contact): Greeting Did you get through on your first attempt? (webchat) Was contact maintained throughout the enquiry? (webchat) Did the greeting include “hello (or similar) Connexions Direct? (telephone) If you were put on hold, did the adviser explain what they were doing? (telephone) Were you transferred to another adviser within the contact centre? (telephone) If you were transferred, was your permission sought before you were passed on? (telephone) If you were transferred, were you informed what was happening? (telephone) Dealing With Your Enquiry Did the adviser suggest phoning to discuss your issues? (text) Were additional webpage(s) used during the webchat? (webchat) If additional webpages(s) were used, were you able to view them? (webchat) If additional webpages(s) were used, were these helpful? (webchat) In completing questionnaires, mystery shoppers were reminded that contacts do not always conclude with a solution to a problem or issue, but should leave the young person feeling that they are better equipped to make a decision or move forward in relation to an issue given the additional information, support and advice offered. Mystery shoppers were also asked to take into account the different forms of contact in making their assessments. Assessment of SMS text services raises particular issues. Our pre-fieldwork consultation suggests that the 160 character limit on SMS contacts encourages CXD Advisers to invite texters to contact the service via phone or webchat, so that they can offer more detailed advice. There is a sense in which the limits of SMS text also impose constraints on what can reasonably be offered by services using this mode. The methodological issue for us was around reconciling on the one hand, a practical recognition of the very different character of each mode of contact (particularly text) and on the other, the need to establish a project which would allow an objective comparison across modes of contact (and implying the creation of standard assessment processes regardless of mode of contact). In the event, this pilot probably achieved something between these two poles (see results below). However, future mystery shopping of the Connexions Direct Service needs to build more systematically on other market research which explores young people’s expectations about each Connexions Direct mode of contact. The latter would usefully inform this difficult issue. Copies of the questionnaires can be found in Appendix B.


TNS © 2004 16

4. Post-Fieldwork Consultation

4.1 Overview Following fieldwork, young people and Personal Advisers from Essex, Southend and Thurrock Connexions Partnership were engaged in further consultation. The purpose of this exercise was to gain additional views on: a) the questionnaires used for assessing mystery shops (in particular to explore what these young people and Personal Advisers regarded as the absolutely essential questions required to allow an effective assessment of the Service); b) the results obtained (in terms of whether the substantive advice offered to shoppers was ‘appropriate’ in relation to individual scenarios). In relation to b), a concern to achieve an ‘external’ validation of results, particularly from relevant professionals, reflects a recognition that the mystery shoppers themselves are not in a position to pass total judgement on the content of contacts with Connexions Direct. An important caveat is also warranted at this stage. The pilot research programme did not allow for an extended period of post-fieldwork consultation. Future such initiatives would need to build on what was achieved in the pilot. In relation to objective b) in particular, further work would be needed to better inform assessments of the ‘validity’ of responses given by Connexions Direct Advisers. It perhaps goes without saying though that these kinds of validation exercises will tend to operate at a general level given the number and variety of shops. 4.2 Questionnaires Young people and Personal Advisers were asked for feedback on which of the questions they saw as more or less important, by ranking them from one of the following: 1. Essential – the adviser must do this 2. Preferred – it would be good if the adviser did this 3. Not needed – the adviser would not be expected to do this Given timing constraints, this post-fieldwork ‘prioritisation’ exercise was limited to a review of the questionnaires designed for telephone shops. A future mystery shopping programme would need to extend this approach to all modes of contact. It should then use the views expressed to weight the mystery shopping results to inform the conclusions drawn about service performance. Future work exploring the relative significance of individual questions in assessment sheets could reasonably be incorporated into pre-fieldwork consultation as part of the more general review of shopper assessment criteria. In the event, post fieldwork consultation with young people and Personal Advisers in Essex did reveal broad agreement with the range of questions included in the questionnaire (perhaps not surprising given pre-fieldwork consultation). However, one question in particular was viewed by both Personal Advisers and young people as unnecessary and in their view, offered little to support an assessment of the level of service offered: ‘Did the adviser ask you where you had heard about Connexions Direct?’ Given the limited nature of post-fieldwork consultation to explore the relative significance of assessment questions, this pilot study does not attempt to use the information gained to inform a weighting of mystery shopping results. Nevertheless future work of this kind should seek to do this. Section 7 does however offer a little more detail on the views expressed by


TNS © 2004 17

Essex young people and Personal Advisers in relation to questions used to assess the Service. 4.3 Responses Following completion of fieldwork, a code frame was developed to capture the main categories of response to the eight scenarios that shoppers received from CXD advisers. Using this coding frame to structure discussions, we also sought the views of Essex young people and Personal Advisers in relation to the content of advice, information and support offered. In short, we asked the question: ‘Did the advice, information or support seem ‘appropriate’, ‘important’ and ‘valid’ given the nature of the particular enquiry (or scenario)?’ The result of this post fieldwork consultation exploring the appropriateness of advice are presented in Section 7. It should be remembered that the feedback came from a group who were both small in number and were arguably more familiar with the local Connexions network than perhaps ‘the average’.


TNS © 2004 18

5. Overview of Results

The following sections look at the results for each mode of contact, each scenario and by individual question from the questionnaires. Whilst the results for text mystery shops are shown along side those for the other contact methods, it should be recognised that text messages are, in the ways outlined above, perhaps the most limited contact method. In terms of substantive advice, information and support given by Connexions Direct staff, the most common response was to offer details of national websites or national telephone helplines (44 per cent). This was the most common response for scenarios 2, “I’m taking my exams soon, but find it difficult to revise” (57 per cent), 5 “I’m worried about my mum who is drinking a lot” (53 per cent) and 6 “My friend is smoking a lot of cannabis” (62 per cent). 34 per cent of mystery shoppers were advised to speak to responsible and relevant adults which, depending on the individual scenario, referred to a teacher, a school nurse, doctor, practice nurse or adult relative. This was the most common response for scenarios 3, “I’ve been asked to choose my options, I just don’t know what to take” (58 per cent) and 7 “My periods are heavy and painful. Is this normal?” (71 per cent). Local information sources, such as libraries and Family Planning Clinics, were given in 31 per cent of contacts. This was the most common response for scenarios 4, “I left school last year, but I don’t know what I want to do now” (68 per cent) and 8, “I had sex with my boyfriend and now I think I might be pregnant” (47 per cent). It should be noted that where ‘local information sources’ are discussed, this does not mean an actual referral to a local Connexions Partnership. Mystery shoppers were given a ‘definitive answer or answers to their question’ (as opposed to being ‘signposted’ or referred to another agency, expert or source) in 26 per cent of contacts. This was the most common response for scenario 1, “I’m thinking about going into nursing after I’ve finished school. What do I need to get in my GCSEs?” (69 per cent). At the same time, 12 per cent of mystery shops were abandoned due to there being no response to emails or texts; in relation to the telephone, being left on hold, receiving a garbled text message response; or in relation to webchat, losing the link.


TNS © 2004 19

6. Results by Contact Type

The following section reviews the results by mode of contact across all scenarios. While there was some variation across modes, perhaps not surprisingly responses were broadly similar. 6.1 Telephone On average, CXD staff answered mystery shop telephone calls in 3.9 seconds. Encouragingly, a clear greeting was given in 98 per cent of calls. Perhaps rather surprisingly though, only 60 per cent of callers were told that the service was confidential, despite guidance to the contrary. Advisers gave their name in 75 per cent of calls and asked for the caller’s name in 56 per cent. 88 per cent of callers were asked their age, while 87 per cent were asked where they live. 21 per cent of callers were transferred to another adviser within Connexions Direct. Permission was sought before the transfer in 97 per cent of calls and everyone who was transferred was fully informed about what was happening to their call. 97 per cent of advisers appeared to listen fully to the caller. The adviser’s tone put 93 per cent of callers at ease and appropriate language was used in 97 per cent of calls. Shoppers felt that a satisfactory response had been offered in 86 per cent of calls and the information given was clear and relevant in 92 per cent of calls. A referral to a local Connexions Partnership was offered in just 1 per cent of cases. Overall a mark out of 10 of 7.74 was given, the second highest mark by contact type. Figure 1 - Most Frequent Responses* (Telephone)

* For all contact modes, more than one piece of advice was possible, for example the adviser could suggest talking to a responsible adult, whilst at the same time recommending a useful website

Top 6 Responses - Telephone

46% 43% 37% 35%19% 15%

0%20%40%60%80%

100%

Offerednational

informationsources

Offered localinformation

sources

Advised tospeak to

responsibleand relevant

adult

Giveninformationabout thesubject

Self helpoptions

Advised tospeak to

person givingcause forconcern


TNS © 2004 20

Figure 1 shows that 46 per cent of telephone contacts resulted in an offer of website addresses and/or telephone numbers for national websites or national telephone helplines, whilst 43 per cent resulted in an offer of local information sources. 37 per cent of callers were advised to speak to a responsible and relevant adult, such as a doctor or teacher and a solution or answer to the mystery shop was given in 35 per cent of calls. Self-help options (such as, and varying according to the individual scenario: breaking up revision, revising with friends, taking painkillers or using a pregnancy testing kit) were suggested in 19 per cent of calls. 15 per cent of callers were advised to speak to the person giving cause for concern (for example the friend using cannabis or the mother with the alcohol dependency). Three per cent of calls were abandoned after the callers were left on hold or no response was received. 6.2 Email On average it took 8.5 hours for CXD advisers to respond to emails. 100 per cent of advisers appeared to have read the mystery shopper’s email and the tone of responses put the mystery shopper at ease in 97 per cent of cases. Information sources were given in 81 per cent of emails and a referral to a local Connexions Partnership was offered in 1 per cent. A satisfactory response was gained in 97 per cent of emails while the information given was clear and relevant in all cases. The overall mark out of 10 for email contacts was 8.11, making this the highest scoring mode of contact. Figure 2 - Most Frequent Responses (Email) Figure 2 shows that the most frequent response to emails was to offer details of national sources (65 per cent), while CXD contact details were given in 50 per cent of contacts. Both of these responses were received significantly more on email compared with other contact methods. Meanwhile, 41 per cent were advised to speak to a responsible and relevant adult, and ‘self-contained’ information and advice (as opposed to signposting or referral to

Top 6 Responses - Email

65%50% 41% 40%

27% 22%

0%20%40%60%80%

100%

Offerednational

informationsources

GivenConnexions

contactdetails

Advised tospeak to


adult


sources


Self helpoptions


TNS © 2004 21

other sources) was given in 27 per cent of emails, with self-help options being offered in 22 per cent of emails. Twelve per cent of emails received no response. 6.3 Webchat Contact was made at the first attempt in 67 per cent of interactive webchats and was maintained in 57 per cent of all webchats. On average it took 76.3 seconds to respond to webchats, while all responses included a greeting. The advisers appeared to have fully read the mystery shopper’s question in 97 per cent of webchats and asked further questions in 82 per cent. Additional webpages were used in 42 per cent of webchats, 91 per cent of which could be viewed successfully. Whenever they were used, webpages were rated as helpful. The adviser’s tone put the mystery shopper at ease in 87 per cent of webchats and appropriate language was used in 99 per cent. A satisfactory response was gained in 83 per cent and the information was clear and relevant in 97 per cent of cases. The overall mark out of 10 was 7.32, placing this mode of contact below telephone and email. Figure 3 - Most Frequent Responses (Webchat) Figure 3 shows that during 33 per cent of webchats, the mystery shopper was advised to speak to a responsible and relevant adult. National information sources were offered in 31 per cent and local information in 25 per cent. Information to answer the mystery shopper’s enquiry –a definitive or self-contained response - was given in 20 per cent of instances. Mystery shoppers did not abandon calls immediately following an initial breakdown in contact (despite breakdowns taking place in nearly half of contacts). Typically they tried to re-establish contact. Nevertheless, 26 per cent of webchats were abandoned either because contact was repeatedly lost or because there was no initial response. One mystery shopper commented “I tried several times for an hour to connect, but without success”, whilst another commented “The adviser seemed very sympathetic…unfortunately we were cut off before ending the chat”.

Top 7 Responses - Webchat

33% 31% 26% 25% 20% 20%

0%20%40%60%80%

100%

Advised tospeak to


adult

Offerednational

informationsources

Callabandoned


sources

Sent oroffered

relevant webpages orhotlinks



TNS © 2004 22

6.4 Text The average time to answer text messages was 49.7 minutes. CXD advisers suggested that the mystery shopper should contact CXD via the website or telephone, to discuss their issue with an adviser in 52 per cent of contacts. Meanwhile, the adviser appeared to have fully read the mystery shopper’s text in 92 per cent of instances and asked further questions in only 9 per cent, perhaps reflecting the nature of the contact mode. The tone used put the mystery shopper at ease in 71 per cent of texts and appropriate language was used in 89 per cent. While the overall mark out of 10 for text contacts was 6.69, making this the lowest scoring form of contact, a satisfactory response was gained in 80 per cent of texts and the information was viewed as clear and relevant in 86 per cent of cases. Mystery shoppers were reminded before they made their shops that texting has clear limitations compared with other modes of contact and that they should remain sensitive to this in their assessments. However, further work is needed to establish the expectations of young people who contact the Service by text and to ascertain whether their expectations do vary according to the mode of contact and indeed, whether they align their chosen method of contact according to the nature of the information, advice and support they require. Figure 4 - Most Frequent Responses (Text) Figure 4 shows that the most frequent response to text messages was to offer national information sources, including the CXD website (37 per cent). The mystery shopper was asked to call (or their friend was asked to call) a given number (either CXD or another relevant number) for advice or to discuss further in 33 per cent of texts, significantly more than on other contact methods. Given the satisfaction rating of 80%, mystery shoppers clearly viewed the suggestion that the shopper call a number as a satisfactory solution. 19 per cent of mystery shoppers were advised to speak to a responsible and relevant adult and 16 per cent were given ‘self-contained’ information and advice (as opposed to signposting or referral to other sources) in relation to their scenario. Texts were abandoned due to no response or an illegible response in 13 per cent of contacts, one such response read “It may be helpful if you could talk this through Z£@£@£@@@~£@£”. Rather surprisingly, advisers responded by saying that they recognised the shopper as calling about the same issue previously in 4 per cent of cases (equivalent to 3 SMS texts).

Top 6 Responses - Text

37% 33%19% 16% 13% 9%

0%20%40%60%80%

100%

Offerednational

informationsources

Asked to callor friend

asked to calla givennumber

Advised tospeak to


adult


Callabandoned


sources


TNS © 2004 23

7. Results by Scenario

The following section reviews the results by scenario and incorporates feedback received from young people and Personal Advisers in Essex, Southend and Thurrock Connexions Partnership. The latter helps to assess the appropriateness of responses received to the mystery shops. 7.1 Scenario 1 - I’m thinking about going into nursing after I’ve finished school. What do I need to get in my GCSEs? In 69% of mystery shops, information was offered on nursing which, as might be expected, was viewed as appropriate by Essex Personal Advisers and young people. However, Essex representatives also suggested that a referral to local Connexions Partnership would be relevant in relation to this scenario, despite it only being offered by Connexions Direct Advisers in 2 per cent of mystery shops. An offer of national information sources was given in 44 per cent of contacts whilst local information sources were given in 36 per cent of cases. Meanwhile, Web pages or hotlinks were offered or sent in 20 per cent of contacts and were rated as important by Essex Personal Advisers and young people. 7.2 Scenario 2 - I’m taking my exams soon, but find it difficult to revise. Can you help me? 57 per cent of mystery shops resulted in an offer of national information sources whilst self-help options, viewed as particularly important by Essex Personal Advisers and young people, were offered in 55 per cent of cases. The mystery shopper was advised to speak to a responsible and relevant adult, usually a teacher, in 42 per cent of contacts and local information sources were offered in 20 per cent of cases. Despite their tendency to be offered only in a minority of cases, Essex young people and Personal Advisers viewed local information sources as potentially helpful in relation to this scenario. 18 per cent of contacts were sent or offered webpages or links to websites whilst a further 10 per cent were given the CXD contact details. Perhaps surprisingly, actual ideas/tips to help with revising were only offered in 17 per cent of contacts. 7.3 Scenario 3 - I’ve been asked to choose my options, I just don’t know what to take. The most common response was to advise the mystery shopper to speak to a responsible and relevant adult, such as a careers adviser, teacher or parent (58 per cent). Post fieldwork consultation with Essex Personal Advisers and young people suggests the appropriateness of this approach. Interestingly, Essex Personal Advisers and young people also viewed referral to a local Connexions Partnership as an important and appropriate response in relation to this scenario. Information about choosing options was given in 41 per cent of contacts. Local information sources were offered in 31 per cent of contacts, the importance of which was also recognised in Essex. Fewer mystery shoppers (just 1 in 5) were offered national information sources which coincided with a view in Essex that such information was less likely to be useful given variability across schools in subject option structures and processes.


TNS © 2004 24

7.4 Scenario 4 - I left school last year, but I don’t know what I want to do now. Local information sources, including details of local Connexions Partnerships, were offered in 68 per cent of mystery shops, whilst national information sources were offered in 34 per cent. At the same time, Essex Personal Advisers and young people argued that referral to a local Connexions Partnership would be appropriate in these circumstances; an approach not suggested by CXD advisers for this scenario. Ideas and advice over what to do now was given in 15 per cent of cases. 7.5 Scenario 5 - I’m worried about my mum who is drinking a lot. National information sources were offered in 53 per cent of contacts, whilst 40 per cent were advised to speak to Mum. Interestingly, Essex Personal Advisers and young people noted that speaking to Mum may be inappropriate in the circumstances given that this might inflame the situation. Again, they argued more firmly in favour of a local referral to a Connexions Partnership Personal Adviser for more sustained contact and viewed details of local information sources as more relevant in this situation than national information sources. (local information sources were given in 20 per cent of contacts). 25 per cent were advised to speak to a responsible and relevant adult, usually the grandmother, whilst Advisers gave definitive ideas and solutions for dealing with the situation in 12 per cent of cases. 7.6 Scenario 6 - My friend is smoking a lot of cannabis. The most common response was to offer national information (62 per cent). Essex Personal Advisers and young people also viewed this as the most appropriate way forward. At the same time, mystery shoppers were advised to speak to the friend in 60 per cent of contacts, though interestingly, young people and Personal Advisers in Essex did recognise that there were risks attached to such a strategy. Local information sources were given in just 18 per cent of calls The mystery shopper was advised to speak to a responsible and relevant adult in only 7 per cent of cases. The small number of instances of this approach being suggested coincides with a view from young people and advisers in Essex that it might be more appropriate to speak to an older sibling or friend, rather than somebody in authority. 7.7 Scenario 7 - My periods are heavy and painful. Is this normal? During 71 per cent of mystery shops, the Adviser suggested that the mystery shopper should speak to a responsible and relevant adult such as parent or doctor and this corresponded closely with the views of young people and Personal Advisers in Essex. National information sources were given in 46 per cent of cases, whilst self-help options were offered in 25 per cent of contacts (the latter was viewed as particularly appropriate by Essex young people and Personal Advisers). 7.8 Scenario 8 - I had sex with my boyfriend and now I think I might be pregnant. The most common responses were to offer local information sources, such the details of the nearest Family Planning Clinic (47 per cent), or to suggest self-help options, such as the purchase of a pregnancy testing kit (47 per cent); both of which were rated as appropriate responses in our consultation with Essex Personal Advisers and young people. 41 per cent of mystery shoppers were advised to speak to a responsible and relevant adult, such as a doctor (also viewed as appropriate in Essex), whilst 36 per cent were offered national information sources (viewed as perhaps less relevant in Essex).


TNS © 2004 25

8. Results by Question

The following section reviews the results attained for ‘common’ questions that feature on at least two questionnaires and compares the results by mode of contact. 8.1 Greeting A clear greeting was given in 100 per cent of webchats, 98 per cent of calls, 96 per cent of emails and significantly less in texts, just 9 per cent (the latter arguably reflecting the limitations of SMS text). The adviser was most likely to give their name on email (95 per cent) and webchats (93 per cent) and significantly less likely on calls and texts (75 per cent of calls and 3 per cent of texts). Post fieldwork consultation with young people and Personal Advisers in Essex showed that a clear greeting was viewed as an essential element of the contact, but the manner of the greeting was also seen as important. People in Essex also agreed that it was essential a) to ask if the service had been used before to ensure continuity in service delivery; b) to explain that the service is confidential and c) for the adviser to give their name. 8.2 Dealing With Your Enquiry Mystery shoppers were significantly more likely to be asked their name during webchats compared with telephone contacts (94 per cent of webchats compared with 56 per cent of telephone calls). The percentage who were asked their age and where they live was not significantly different by mode of contact; on calls, 88 per cent were asked their age whilst 87 percent were asked their location. Corresponding figures for webchat were 92 per cent and 85 per cent respectively. Advisers asked where the mystery shoppers had heard about Connexions Direct during 60 per cent of calls and 50 per cent of webchats, a difference between the modes which is not statistically significant. The exercise in Essex showed that Personal Advisers viewed questions around age and location as essential given the need for information and advice to take into account such factors. Young People and Personal Advisers agreed that there was no need to ask where the young person had heard about CX Direct. Young people in particular did not see it as essential for advisers to request their name during calls, recognising that callers can elect to remain anonymous. The adviser appeared to listen fully to (or read fully) the mystery shopper’s question and subsequent responses in the vast majority of mystery shops, with no significant differences revealed between the contact types (100 per cent of emails, 97 per cent of calls and webchats and 92 per cent of texts). Further questions were asked by the adviser during 86 per cent of calls, 82 per cent of webchats and, significantly less, in 9 per cent of texts. Consultation with Personal Advisers and young people in Essex highlighted, not surprisingly, the importance of CXD advisers listening fully to young people. Information sources were offered in 81 per cent of emails, 66 per cent of calls, 51 per cent of webchats and in 46 per cent of text contacts - significantly less than email and telephone calls. However offers of actual referral to a local Connexions Partnership were only made during 1 per cent of calls and emails and not at all via other contact methods.


TNS © 2004 26

8.3 Overall assessment of contacts Mystery shoppers felt that the adviser’s ‘tone put them at ease’ most often in email (97 per cent), telephone (93 per cent) and webchats (87 per cent) and significantly less so via text (71 per cent). In the view of shoppers, advisers were more likely to use ‘appropriate language’ on email (99 per cent), webchats (99 per cent) and calls (97 per cent), compared to a significantly lower, albeit still relatively high figure of 89 per cent on texts. The mystery shoppers felt a satisfactory response was given to their question in 97 per cent of email contacts, significantly higher than the 83 per cent of webchats and 80 per cent of texts, though not significantly different to 86 per cent of telephone calls. One mystery shopper using the webchat commented that their webchat had been “very professionally handled with lots of useful information, encouragement and advice”. The information was more likely to be perceived as ‘clear and relevant’ via email (100 per cent), webchats (97 per cent) and the telephone (92 per cent); one mystery shopper using email commented “although it took a while to get a response, the information given was useful and informative”. This compared with a significantly lower figure of 86 per cent of contacts by text. However in a minority of contacts, mystery shoppers felt that they were actually given too much information; one mystery shopper commented “the call was long and I was bombarded with information which was a little overwhelming”. Meanwhile, the adviser checked that the question had been answered to the mystery shopper’s satisfaction during 77 per cent of telephone calls, which was not significantly different to the figures of 74 per cent for webchats and 67 per cent for emails. The adviser was however, significantly more likely to remind the mystery shopper that they could always make a further contact if more help was needed in email contacts (91 per cent of emails) compared to 73 per cent of calls and 70 per cent of webchats. Young people and Personal Advisers in Essex viewed, again perhaps not surprisingly, that it was essential that contacts were responded to satisfactorily and that clear and relevant information was offered. They also viewed it as essential to check that the young person's question had been answered to their satisfaction and remind them that they can always contact CXD again. Taking the whole contact into account, email contacts were given the highest mark out of 10 ((8.1), significantly higher than the mark for webchats (7.3) and text (6.7), though not significantly different from the mark for telephone (7.7).


TNS © 2004 27

9. Recommendations

In this section we outline a number of recommendations for future mystery shopping of the Connexions Direct Service based on this pilot exercise. Overall this pilot mystery shopping programme worked well and has produced a good basis for future work. The involvement of young people and Personal Advisers, proved to be invaluable and the questionnaires and scenarios developed could form the basis of future work. 9.1 Mystery shoppers Prior to the start of this pilot, discussions were held over the most appropriate mystery shoppers to use. Whilst there were some arguments for using young people known to Local Connexions Partnerships, we advised against this approach. We were concerned that by using young people outside of TNS, there may be issues over their capacity to complete the work using consistent assessment criteria and according to the carefully planned sample and short timetable. We also questioned whether the expectations of young people who were already familiar with their Local Connexions Partnerships, may be different to the ‘average’ user of the CXD Service. At the same time however, we recognised the important need to match the characteristics of shoppers to service users to ensure credibility and realism. As such we recommended:

a) that only existing mystery shopping-trained TNS employees should be used; and that these should be from the youngest age group (in our case 18-21 years); and that

b) a smaller number of younger shoppers (aged 14-17 years in our case) who were children of existing TNS staff should also be used. The latter were subject to close supervision (as described earlier).

There seems no fundamental reason why this kind of hybrid approach could not work in future mystery shopping exercises. We would also recognise the potential value of developing a more sustained relationship with an individual local Connexions Partnership to explore the possibility that they provide a supplementary source of young people who might be trained in mystery shopping techniques and undertake some of the fieldwork. This would require time and resource commitment from the local Connexions Partnership, DfES and from the research contractor to ensure careful management and achievement of an objective and transparent approach; but it would offer the advantage of increasing the pool of ‘young people’ mystery shoppers, as well as provide development opportunities for the young people concerned. We would of course also recognise the crucial role of young people and indeed local Connexions Partnership Personal Advisers (together with Connexions Direct staff) in the design of the project (including scenarios and assessment criteria) and, following fieldwork, in the analysis of the appropriateness of responses received. A more structured relationship with one or two local Connexions Partnerships would also be beneficial here. Elsewhere, there is some evidence to suggest that shoppers were more persistent than ‘real’ users. In particular we would speculate that many more attempts to get through on the website were made by shoppers than would be the case by the ‘average’ user. At the same time, finding mystery shoppers to complete the work over a weekend proved time consuming and highlighted that many young people work on a Saturday. The results from other/future market research projects examining the expectations of users of the Service should be


TNS © 2004 28

drawn upon to inform the development of an ongoing mystery shopping programme to ensure that the mystery shopping reflects the behaviour of real users of the Service. To summarise, the expectations and views of young people and Personal Advisers should be drawn on through a combination of further market research and consultation achieved through a more structured relationship with at least one Local Connexions Partnership. Such a relationship may also provide a supplementary source of ‘young people’ who might be trained to undertake a proportion of mystery shops alongside existing research contractor staff (and their children). Drawing on a number of sources for mystery shoppers (local Connexions Partnerships, existing research contractor staff and the children of older staff) would allow the researcher to balance the arguably greater risks around using young people with no ‘formal employment link’ to the researcher with the obvious benefits of using young people to undertake ‘realistic’ shopping. 9.2 Scenarios The range of scenarios gave a good spread of issues and reflected the type of enquiries received by the service. They also reflected the higher use of the service by females. Eight scenarios provided a good range, but also allowed for the results to be easily examined. Future work should use the same broad categories of scenario to allow for tracking over time, with minor amendments to reduce the possibility of recognition. Scenarios should however be reviewed to ensure that they continue to reflect patterns of enquiry. There is some anecdotal evidence to suggest that the advisers recognised the scenarios used during the mystery shopping. At the same time, a very small minority of responses to texts stated that the mystery shopper had already contacted the Service with a similar scenario and they should call the Service if they needed further assistance (some mobile numbers were repeatedly used, reflecting an earlier understanding that CXD Advisers did not have access to numbers). The extent of the problem of scenario recognition is difficult to gauge as some advisers may have recognised the text number, rather than the circumstances of the scenario. However, and assuming that there was in fact some CXD adviser recognition of scenarios, future fieldwork should be spread over a longer period than the four weeks used in the pilot, to reduce the possibility of recognition. 9.3 Assessment Questionnaires The Essex exercise revealed that more consultation work will be needed to ensure that the questionnaires capture all relevant aspects of the CXD performance. This consultation could also be used to prioritise questions (and their response) to enable weightings to be applied to each question at the reporting stage to reflect their perceived relative importance. As suggested earlier, the variation in expectations when using different contact types should also be examined. It is likely, for example, that expectations will be different when using text as a mode of contact compared to telephone. Similarly, young people may adapt their contact method to reflect the nature of their inquiry and again, further market research might be helpful here. 9.4 Reporting The low base numbers for some contact types at certain times of day, made it difficult to draw conclusion about the results. For this reason, findings are not broken down by time of day and day of the week in this report. Further discussion is needed over whether results are required by day and time slot. If these results are viewed as necessary to an overall assessment of performance, sample numbers may have to be increased to ensure that results are statistically significant.


TNS © 2004 29

The Department should also consider the use of these and future of findings to support the performance monitoring of Connexions Direct customer service. Ideally mystery shopping should be combined with other methods, perhaps drawing on the views of Service users, to make an overall assessment of service quality. However, as reiterated throughout, future work would need to consider a more systematic basis for assessment of the substantive content of CXD adviser information, support and advice. Feedback from future mystery shopping should also be given to CXD adviser staff on their performance to support future staff and service development; though with no requirement to highlight the practice of individual advisers. Although the purpose of the pilot was largely to develop and test an effective programme of mystery shopping, the results attained from this pilot will be valuable. Overall, some excellent practise was revealed, but areas for improvement were also highlighted, both in terms of how the enquiry is dealt with and the actual advice that was given. More generally, the pilot exercise has offered important methodological lessons for the future implementation of mystery shopping in relation to this Service.


TNS © 2004 30

10. APPENDIX A –Briefing Notes

You should not conduct any mystery shops until you have received a full briefing from Emma Starling. The front page of your questionnaire will tell you what time and date you need to start your mystery shop. Do not start you mystery visits at a different time/date as other mystery shops will be taking place. The front page will also show you which scenario you need to use. The details of the scenario are listed below, along with some alternative wording that you could use and additional information that you may need to give to follow up questions from staff. Guidance is offered on how the opening line would be presented on text, email or webchat, but do feel free to use alternative wording. We’ve listed some possible follow up questions, so that you can prepare before the call. Under each opening line is a short synopsis to help you ‘get in to character’. It is unlikely that these notes cover all possible questions from the adviser, so you will need to think carefully about the issue before you call. Remember that you are supposed to be between 13 and 19 years of age, so you need to think about how you would have felt about the issue you are contacting Connexions with. You should know your date of birth in case you are asked for it – please calculate a date of birth before starting your mystery shop. You should also think of a town where you live and know the first part of your postcode. You won’t need to have a full address and you can decline to give an address if asked for it. Learning and Career Planning Scenarios Scenario 1 – I’m thinking about going into nursing after I’ve finished school. What do I need to get in my GCSEs? You know very little about what routes you can go down and so you just want some general pointers. You have heard somewhere (not sure where) that you can get a degree in nursing. You are 16 and in the final year of your GCSEs. You’re hoping to get 8 GCSEs (grades A-C). You’re planning to do A-levels at school. You hadn’t thought about any studying beyond A-levels. You like science at school and plan to do A-level biology. You haven’t decided which other A-levels to do.

Alternative wording: What GCSEs do I need to go into nursing? I’m interested in nursing, what grades do I need in my GCSEs? What sort of grades do I need in my GCSEs to be a nurse? On txt/webchat: I’m finkin of goin in 2 nursin – wot GCSEs wud I need? Follow-up questions/approach from the Advisers: How old are you? What grades are you expecting? Do you know all the different routes you could go down to get into nursing? How much do you know about it already? What makes you want to go into nursing? Advisers will want to explore whether you’re being ‘realistic’ about what’s needed (in terms of the range of training) to get into nursing? They’ll point you in the direction of websites and possibly encourage you to speak to a Connexions Personal Adviser. They’ll be giving you lots of information sources – giving you all the materials so that you can go away and think about it. Scenario 2 – I’m taking my exams soon, but find it difficult to revise. Can you help me? You are 14 years old and are studying for your SATS which start in May (subjects include: English, Maths, Mental Arithmetic, Science). You’re worried that your friends seem able to revise, but you can’t. You are trying to revise most nights, but never seem to be able to remember what you’ve revised. You revise sitting on your bed. The house is quite noisy and you can hear your family watching TV downstairs. You’ve got a younger brother (13) who makes lots of noise in the house.


TNS © 2004 31

Alternative wording: I’ve got exams coming up and I find it difficult to revise/don’t really know how to revise I’ve got exams soon and I need some help sorting out revision. On txt/webchat: Hi I'm takin me exams n i findin em difficult 2 revise 4. Can u help me? Follow-up questions/approach of the Advisers: Which exams are you doing? When is your first exam? Where you’re revising? For how long? What approach are you using at the moment? What are you finding particularly difficult? Which subject are you most worried about? Have you thought about somewhere you could go to revise, perhaps somewhere a bit quieter?' (e.g. friends house, school, library?) They’ll be wanting to comfort you/ offer reassurances -get you to think about what you could do to change the way you revise and, if that’s part of the problem, the environment in which you’re revising. Scenario 3 – I’ve been asked to choose my options, I just don’t know what to take? You are 14 (year 9 of school). You like art, but your Mum thinks it’s not a good subject to study. Your mum wants you to do French/Geography/History (pick one), but you hate it. Alternative wording: I’d got to choose my options at school, but I don’t know where to start. On txt/webchat: I don’t know wot options 2 tak Year 10 - Year 11 Core compulsory subjects are: Citizenship (leading to half a GCSE) English (English Language and Literature leading to 2 GCSEs) ICT short syllabus (leading to half a GCSE) Mathematics (leading to one GCSE) PE (no qualification) RE (leading to half a GCSE) Science Double award covering biology, chemistry and physics (leading to 2 GCSEs) Options - Block A Students are allowed to select one of the subjects below to study in more depth GCSE ICT - Business and Communication systems (1 GCSE) GCSE ICT - Information and Communication Technology (1 GCSE) GCSE Applied ICT - Applied Information and Communication Technology (2 GCSEs and for more able students only) Options Block B Students are allowed to select three subjects below. Some subjects lead to more than one GCSE and where that is the case they have to count them as more than one option

• Asdan Awards/key skills - for students who would struggle to complete GCSE subjects - covers skills such as writing letters, speaking to adults confidently, managing money, using timetables etc.

• College Taster course - designed for less motivated students who find the school curriculum too challenging - involves day release at college to study catering, childcare etc.

• GCSEs:

Art and Design Design and Technology: Product Design Drama

French Geography History

Media Studies Music Spanish


TNS © 2004 32

Science Triple award modular (counts as 3 option choices) ICT

Applied Health and Social Care (Double award) - counts as 2 option choices

Applied Leisure and Tourism (Double award) - counts as 2 option choices

• NVQs: French Spanish Follow up questions/approach of the adviser: Actually it seems like you’ve got a lot of choices and it can look pretty scary, but lets look at it one step at a time (i.e. they will try to calm you down) Which /How many subjects have you got to take? Which are set options? (refer to compulsory subject listed above)? OK so you’re left with X number of options – what sorts of things do you enjoy doing? What we say is choose the things you enjoy…. Have you got any careers in mind? Chat to your mum and dad about it Is there someone in your school you can talk to about it? Have you spoken to your teacher? Scenario 4 – I left school last year, but I don’t know what I want to do now. You left school after your GCSEs. You’ve got 3 GCSEs (English language C, geography C and art B). You’re 17. You don’t want to do any more study as you didn’t like school. Since leaving school you’ve been helping out your Dad with admin at his office, but it’s boring. Pick one of the following if you are asked if you’ve got any ideas/anything you enjoy doing: I like children – I look after my nieces quite often I might like to work with children Or: I like fashion I go shopping a lot with my friends I might like to work in a clothes shop – something like a manager though – not just a shop assistant Or: I like computers and computer games Alternative wording: I left school last year, but don’t know what to do now. What sort of thing could I do? On txt/webchat: Hi I left school last year n i don't no wot i wanna do. Cud u help me? Follow up questions/approach of the adviser: Did you get any qualifications? What grade were they? What subjects were they in? What did you get in maths? Have you considered retaking maths? Are you working or doing some training at the moment? Did you want to start working? Or do you think you’d want to do another course, carry on with your education? Anything you particularly enjoy doing? What kind of business is your Dad in? Family and Relationship Scenarios Scenario 5 – I’m worried about my mum who is drinking a lot. You are 15 and live with your mum, Nan and younger brother (9). Mum is getting drunk about 4 times a week. She’s drunk rather than tipsy and you and Nan have had to put her to bed. She always comes home, but this/last weekend she stayed out all night. Your Mum usually looks after you when she is not drinking. She only gets angry/moody with you when you fight with your brother, but is never violent or abusive. Your Nan does a lot around the house and is always around to cook your tea, but is too old to do the hoovering. You’ve talked to Nan, but she says mum is just upset. You haven’t spoken to anyone else. You want to be able to help her stop drinking. Alternative wording: My mum has started coming in wrecked all the time.


TNS © 2004 33

On txt/webchat: Hi I'm rly worried bout my mum. She keeps drinking lots. Who can I tlk 2? Follow up questions/approach of the adviser: They’ll try to establish if you’re being looked after properly. Who do you live with? Who looks after you? What is her behaviour like/Does it change the way she behaves? How much does she drink? Is she ever violent or does it make her withdrawn/not bothered about anything? If live with dad/other adult - does he drink as well? Are other family members worried about mums drinking? How long has the drinking been going on for? Why does Nan say Mum's 'just upset'? Is the drinking linked to another event e.g. partner leaving etc? Does the drinking worry your mum? Is she aware she's drinking too much? Have you talked to anybody else about this? Can you talk to your mum about this? Could you explain to your mum how worried you are about her drinking? Is it affecting other things in your mum’s life e.g. work or friends? What does Mum do for a living? Scenario 6 – My friend is smoking a lot of cannabis. You and your friend are 17 and in full time education. You both started at a new college in September, although you are doing different courses, and have a lot of new friends. Your friend is from school and you think he/she has started smoking cannabis to fit in with college friends. You are worried as your friend is an asthmatic. He/She keeps saying that it’s safer to smoke weed than drink. Your friend has started being secretive and excluding you because you won’t smoke, you don’t like the way they have changed. He is really boring when he is stoned and just sits there giggling or ignoring you. You would like to know how you can persuade him to stop. His mum and dad either don’t know or care. You have made quite a few friends on your course, so it’s not as if your real concern is that you're been left on your own, it’s just that X is a good friend and you've known them since school, and don't want to lose them. Alternative wording: You could use alternative words for cannabis. On txt/webchat: my friend is smokin a lot of weed and I’m worried bout er/im Follow up questions/approach of the adviser: What courses are you both doing? Are you and your friend enjoying college? Does he know it's illegal? Discuss that cannabis although reclassified is still illegal and slows down reactions. Does the friend realise he's changed? Is he oblivious to the effect it is having on the people around him? Does he smoke alone? How much does he smoke? Ask the young person how has his friend changed, has he become moody, irritable or less trustworthy Discuss with the young person how they want to deal with their friend and offer coping strategies Give details of ask frank website General and Sexual Health Scenarios Scenario 7 – My periods are heavy and painful. Is this normal? You are 14 and live at home with mum and dad. You have a good relationship with dad but constantly row with mum. You’ve been to the doctors, but the doctor told you to take ibuprofen when you get the pain. You felt that they did not take her seriously because of your age. Your periods started two years ago. You would like to know if the pain will last for ever. The last one lasted for 10 days and you had to take a couple of days off school because you felt so ill. Your back hurts a lot.


TNS © 2004 34

Alternative wording: On txt/webchat: my periods r heavy n hurt. Is di normal? Follow up questions/approach of the Adviser: How old are you? Have you been to the doctors? Have you spoken to your mum/friends? How long do the periods last? What sort of pain is it? Do you take any painkillers, for example paracetamol? What do you and mum keep rowing about? Scenario 8 – I had sex with my boyfriend and now I think I might be pregnant. You are 17 and had sex for the first time with your boyfriend about 3 weeks ago. You usually have fairly regular periods and are about 4 days late. You cannot remember being more than a couple of days late before. You haven’t taken a test. You don’t want to talk to your parents about it, nor would you be happy going to your local doctor in case your Mum found out. You’ve mentioned it to your boyfriend and he is scared too. Alternative wording: I think I might be pregnant. On txt/webchat: I ad sex wit my b/f I fink I’m pregnant Follow up questions/approach of the Adviser: When was your last period? Are they normally pretty regular? Have you used a pregnancy test at home? Do you know where to go to get a free pregnancy test (e.g. Young Person’s clinic)? I’ll tell you where your nearest one is…. Can you talk to your boyfriend about it? Or your mum? In other words, highlighting possible sources of support… The adviser will want to double check whether sex happened within the last few days - and whether 'morning after' pills are still a possibility General Instructions For all mystery shops, you should ask your initial question, then allow the member of staff to lead the conversation. They will do this by asking you further questions. Do not volunteer all the summary information contained in the scenario, this should be offered in response to follow up Personal Adviser questions/general conversation with Personal Advisers. If you are asked if you have used Connexions before, say no. Webchats, emails and texts are a more direct form of communication, whilst on telephone calls, the callers are often more reticent with the initial enquiry and any subsequent responses, particularly for the health and relationship questions. The young person will want to feel that they are able to trust the adviser first. Remember that you will be more familiar with the questions staff are likely to ask than the actual users of the service are. It is essential that you sound convincing. You should “um” and “err” as most people would not answer questions straight away. They won’t be surprised by periods of quiet as you deliberate. The advisers will want to explore how you feel about the issue you are contacting them with. Advisers will present options and then invite you to offer your thoughts on what you have heard. Often, contacts do not conclude with a solution to the problem, but they should send you away with information to allow you to make a decision or take the next step. On your original email and SMS do not put your name. On your email just finish it with ‘cheers’ or ‘thanks’, on your SMS, just ask your enquiry with no initial greeting or parting message. This gives the adviser the chance to ask for your details. Initially just give a first name when the adviser asks, only give your full name if you are asked for it. For the health and relationship scenarios, you could


TNS © 2004 35

say that you don’t want to give your full name at this stage. Advisers may also ask for further background (e.g. age, area you're calling from). Don't volunteer too much info at the outset of telephone calls -let the adviser ask you... Advisers are trained to be supportive, empathetic and they should make you feel comfortable, respected and valued. Their role is not to act as a friend to the enquirer. Advisers may give you details of other agencies you can contact. If this happens, give a yes at the question asking if the adviser offered any local information or information sources. To get a yes to the subsequent question about referring you to another service, they will ask if they can pass your details on to another agency, for them to then contact you. You should decline this invitation -saying that you will contact the agency (or whatever) yourself, you just need to know who to phone. Telephone Calls Dial 141 before you make your call. Please remember that all calls must be recorded. Email If the member of staff asks you if they could contact you to discuss your problem or suggest that you call them, please decline this offer. Do not spell check the email before you send it – the advisers are used to receiving emails with typos. Advisers often receive messages with smileys included in the email. All email correspondence must be printed and stapled to your questionnaire. SMS Dial 141 before you make your call. If the member of staff asks you if they could contact you to discuss your problem or suggest that you call them, please decline this offer. When completing the questionnaire, please bear in mind the limits of text as a mode of contact. The responses from the adviser will be short and to the point. Webchat If the member of staff asks you if they could contact you to discuss your problem or suggest that you call them, please decline this offer. Don’t worry if your webchat contains typos, this will mirror the webchats that advisers deal with. You may also include smileys during the webchat. At the end of the webchat, please print the full conversation and staple it to your questionnaire. Language to use: OK Alright Cheers Yeh Yep Dunno Kinda Um Err Cool


TNS © 2004 36

11. APPENDIX B – Questionnaires

EMAILGreeting Yes No NA

Q1 How quickly did you receive a response to your email? hoursQ2 Did the email include a greeting?

Q3 Did the adviser explain that Connexions Direct is a confidential service?

Q4 Did the adviser give their name?

Dealing With Your Enquiry Yes No NA

Q5 Did the adviser appear to have read your email fully?

Q6 Did the adviser offer any local information or information sources, e.g. localConnexions Partnerships?

Q7 Did the adviser refer you to another service e.g. local Connexions Partnerships orother local service?

What response(s) did the adviser give to your enquiry?

Overall Yes No NAQ8 Did the adviser’s tone put you at ease?

The adviser should make you feel valued and encourage you to continue with your enquiry.

Q9 Was appropriate but professional language used?

Q10 Were you given a satisfactory response to your question?

Q11 Was the information clear and relevant?

Q12 Did the adviser check that your question had been answered to your satisfaction?Q13 Did the adviser remind you that you could always contact Connexions again if you

need further help?Q14 Taking the whole assessment into account, what mark out of 10 would you give this

assessment?Please comment on the answers you have given, in particular explaining any NO answers or areas thatrequire further explanation. Please comment even if you have answered YES to every question


TNS © 2004 37

TELEPHONEGreeting Yes No NA

Q1 How quickly was the call answered? secsQ2 Was the greeting clear?

Q3 Did the greeting include “Hello (or similar) Connexions Direct”

Q4 Were you asked if you had used Connexions Direct before?

Q5 Did the adviser explain that Connexions Direct is a confidential service?


Q7 If you were put on hold at any point, did the adviser explain what they were doing?

Q8 Were you transferred to another adviser within the contact centre?

Q9 If you were transferred, was your permission sought before you were passed on?

Q10 If you were transferred, were you informed what was happening?


Q11 Did the adviser ask your name?

Q12 Did the adviser ask your age?

Q13 Were you asked where you live?

Q14 Did the adviser ask you where you had heard about Connexions Direct?

Q15 Did the adviser listen fully to your question and subsequent responses?

Q16 Did the adviser ask you further questions about your query?Q17 Did the adviser offer any local information or information sources, e.g. local

Connexions Partnerships?Q18 Did the adviser refer you to another service e.g. local Connexions Partnerships or

other local service?What response(s) did the adviser give to your enquiry?



Q20 Did the adviser use appropriate language, whilst remaining professional?



Q23 Did the adviser check that your question had been answered to your satisfaction?


TNS © 2004 38

WEBCHATGreeting Yes No NA

Q1 Did you get through on your first attempt?

Q2 Was contact maintained throughout the enquiry?

Q3 How quickly did you receive a response? secsQ4 Did the response include a greeting?



Q6 Did the adviser ask your name?

Q7 Did the adviser ask your age?

Q8 Were you asked where you live?

Q9 Did the adviser ask you where you had heard about Connexions Direct?

Q10 Did the adviser appear to have read your message fully?



other local service?Q14 Were additional webpage(s) used during the webchat?

Q15 If additional webpages(s) were used, were you able to view them?

Q16 If additional webpages(s) were used, were these helpful?

What response(s) did the adviser give to your enquiry?






Q21 Did the adviser check that your question had been answered to your satisfaction?Q22 Did the adviser remind you that you could always contact Connexions again if you

need further help?


TNS © 2004 39

SMS TEXTGreeting Yes No NA

Q1 How quickly did you receive a response to your text message? minsQ2 Did the response include a greeting?


Dealing With Your Enquiry Yes No NAQ4 Did the adviser suggest phoning you/or you phoning them to discuss your issues in

more detail?Q5 Did the adviser appear to have read your text fully?



other local service?What response(s) did the adviser give to your enquiry?





Q12 Was the information clear and relevant?Q13 Taking the whole assessment into account, what mark out of 10 would you give this

assessment?Please comment on the answers you have given, in particular explaining any NO answers or areas thatrequire further explanation. Please comment even if you have answered YES to every question


TNS © 2004 40

12. APPENDIX C – About TNS Mystery Shopping

TNS Mystery Shopping is the product specialist of TNS, the world’s third largest market information company. We have been providing mystery shopping programmes to a variety of industry sectors for more than 10 years. We have considerable experience of working with a wide variety of organisations in both public and private sectors. Our role is to assist our clients improve standards at the customer/staff interface. We provide unrivalled expertise, the highest levels of data quality and the most advanced reporting tools available. The measurement we provide, and the practical recommendations we deliver, empower our clients to drive service performance throughout their organisations. The commissioning of a mystery customer programme is only the beginning of the service improvement process. Mystery shopping must become a measurement for action at all levels throughout the organisation. TNSMS has identified four critical factors that constitute successful mystery shopping, and, ultimately, the delivery of excellent customer service. • Development of Measurement Standards Our focus is the objective measurement of service at the point of delivery allowing us to provide actionable data to clients. The messages being delivered should complement key service action points. Data must be understandable and actionable, at every level. • Data Collection & Data Quality Our commitment to delivering data quality is reflected in our approach to shopper training and briefing. We will always recommend the correct level of shopper training based on a client’s specific needs. The scenarios used should be credible and • Reporting The communication of performance to programme participants is an essential component of service improvement. The TNS commitment to IT and data delivery systems allows our clients’ access to the most advanced data reporting, data dissemination and data analysis tools available. • Communication The delivery of service messages can be as important as the message itself. We invest considerable time in communicating the benefits of the message to participants.

Copies of this publication can be obtained from:

DfES PublicationsP.O. Box 5050Sherwood ParkAnnesleyNottinghamNG15 0DJ

Tel: 0845 60 222 60Fax: 0845 60 333 60Minicom: 0845 60 555 60On-line: www.dfespublications.gov.uk

© TNS 2004

Produced by the Department for Education and Skills

ISBN 1 84478 287 5Ref No: RR561

www.dfes.go.uk/research

mystery shopping pilot evaluation study - digital education

Documents