the bar standards board central examinations board · the bar standards board central examinations...

1

THE BAR STANDARDS BOARD

CENTRAL EXAMINATIONS BOARD

CHAIR’S REPORT JULY 2013

Executive summary

The Central Examination Board (‘CEB’) has now completed its second cycle of overseeing first sit assessments in the three knowledge areas of the Bar Professional Training Course (‘BPTC’).

The post scaling outcomes of the 2013 first sit centralised assessments following review of cohort performance by the CEB are as follows:

2013 first

sitting* 2012 first

sitting* Change

Professional Ethics Number of candidates 1722 1591 +131

Passing MCQ 94.3% 92.6% 1.7%

Passing SAQ 89.5% 88.5% 1%

Passing overall 86.4% 84.9% 1.5%

Criminal Litigation, Evidence and Sentencing

Number of candidates 1719 1569 +150

Passing MCQ 88.9% 88.7% 0.2%

Passing SAQ 69.9% 77.8% -7.9%

Passing overall 68.2% 74.7% -6.6%

Civil Litigation, Evidence and Sentencing

Number of candidates 1768 1568 +200

Passing MCQ 73.2% 78% -4.8%

Passing SAQ 61.5% 73.5% -12%

Passing overall 56.2 % 68% -11.8%

(*Although the first sit assessment, a number of candidates will have undertaken the assessments in March and April 2013 on a deferred or referred basis.)

The first sit all Provider post scaling passing rates for 2012/13 show very little change from last year in Professional Ethics (indeed a slight improvement) as against a declining performance in Criminal Litigation SAQs and in both Civil Litigation assessments. A more detailed analysis is contained in the main report. .A number of general points should be borne in mind when considering the results data:

Knowledge area assessment passing rates in the above table indicate the aggregated percentage of candidates passing an assessment across all Provider institutions.

All data set out in the body of the report has been presented so as to preserve the anonymity of Provider institutions.

2

There is no pre-scaling data for 2010/11 as this was pre-CEB and all knowledge area assessments were set and marked by individual Provider institutions. Where data in this report is described as ‘post-scaling’ for 2010/11 it is the data supplied by the Provider institutions to the BSB for that year following their final examination boards, and for 2011/12 and 2012/13 it is data showing the effects of any CEB scaling interventions.

1. Context

1.1 The 2010/11 academic year saw the first round of assessments under the BPTC regime (replacing the BVC) in the wake of the Wood Report (July 2008). For 2010/11 all providers were required to assess candidates in Professional Ethics, Civil Litigation, Remedies & Evidence, and Criminal Litigation, Evidence & Sentencing (often referred to as the ‘knowledge areas’) by means of MCQs and SAQs. Together these three modules represent 25% of the BPTC (i.e. 30 credits out of 120). For 2010/11 the knowledge area assessments were set and marked by the Providers. Centralising these assessments was a key recommendation of the Wood Report, and the Central Examinations Board (‘CEB’) was established to oversee this change on behalf of the Bar Standards Board (‘BSB’). 2011/12 was the first year of operation for the system of centralised examinations for the knowledge areas on the BPTC. No changes were made to the format of assessment, but the setting of the assessments was undertaken independently of the Providers by a team of CEB examiners appointed by the BSB.

1.2 In each knowledge area candidates are required to attempt a multiple choice question (‘MCQ’) test, and a short answer question (’SAQ’) test. In the Civil and Criminal papers the MCQ comprises 40 questions and the SAQ 5 questions. In Professional Ethics the MCQ comprises 20 questions and the SAQ 3 questions. All questions are compulsory and the pass mark in each paper is 60%. The marks for the MCQ and SAQ papers are combined to provide a candidate with an overall mark in each of the knowledge areas. Candidates have to achieve the pass mark of 60% in both elements of assessment, there being no scope for aggregation between MCQ and SAQ scores.

1.3 The assessment process is overseen by the CEB whose members are appointed by the BSB. The CEB comprises a Chair, teams of examiners (a Chief Examiner and one or two Assistant Examiners for each knowledge area), an independent observer, an independent statistician and senior staff from the BSB. The Chair and the examiners, who are all independent of the Provider institutions, contribute a mix of academic and practitioner experience.

1.4 From the academic year 2011/12 onwards the CEB has had responsibility for setting the knowledge area assessments and confirming the knowledge area cohort marks for each of the Provider institutions. In confirming marks for cohorts of candidates the CEB is concerned to ensure that a consistent measure of achievement has been applied across all Providers, and that proper account has been taken of any relevant factors that may have had a bearing on the performance of a cohort of candidates. As a result the CEB has the discretion to scale cohort marks (upwards or downwards) if it feels there are issues relating to all candidates, or a statistically relevant sub-cohort of candidates, that justify such intervention. In confirming the marks for MCQ and SAQ papers the CEB does not address the combined effect of the marks in terms of whether or not a candidate has passed a knowledge area overall, or where a candidate falls in terms of grade boundaries.

1.5 Once the CEB has confirmed the MCQ and SAQ marks for each cohort of candidates at each Provider the marks are distributed to the Providers where they feed into the individual BPTC student profiles considered at the Provider award and progression examination boards. It is at the Provider examination boards that issues relating to individual candidates such as extenuating circumstances or academic misconduct are considered.

1.6 There are currently 9 Provider institutions offering the BPTC across 11 centres:

BPP – London and Leeds

Cardiff

3

City University Law School

University (formerly College) of Law – London and Birmingham

Kaplan (in association with Nottingham Trent Law School)

Manchester Metropolitan University

Nottingham Trent Law School

UNN

UWE

All Provider cohort results contained in the report have been anonymised. Providers are identified by numbers 1 to 11 in the graphs of cohort performance featured in this report. The allocation to a Provider of a number from 1 to 11 will vary from one graph to another depending on the baseline adopted for the comparison Provider cohort performance.

1.7 Terms used in this report:

“All-Provider” refers to the aggregated data bringing together cohort performance across all 11 providers

“By Provider” refers to data showing the performance of each of the 11 Providers compared with each other

“First sit” refers to the March/April exam cycle – note that some candidates undertaking these examinations may be doing so on a referred or deferred basis

“Resit” refers to the August/September exam cycle – some candidates undertaking these examinations may be doing so on a deferred basis.

“Combined” refers to the combined MCQ and SAQ performance. Combined passing rates can be lower than either MCQ or SAQ passing rates depending on the pattern of candidate failure.

2. The setting and conduct of assessments

2.1 All Provider institutions were required to supply their 2010/11 knowledge area MCQ and SAQ papers (first sit and resit) along with solutions, to the BSB. The supplied material was used by the CEB examiner teams, with necessary amendments and additions, to devise the 2011/12 first sit and resit assessments and related solutions. Additional material was supplied by the Provider institutions for the 2012/13 assessments, and this was further supplemented by questions and solutions devised by members of the CEB examining teams. Assessment scrutiny meetings were attended by relevant CEB examiners, the CEB Chair and BSB staff. Assessments were also checked by independent technical readers and any issues raised referred back to the scrutiny meetings.

2.2 Candidates at all Provider institutions attempted the assessments in each of the knowledge areas on the same dates.

Professional Ethics – 18th March 2013

Civil Litigation, Remedies & Evidence – 3rd

April 2013

Criminal Litigation, Evidence & Sentencing – 5th April 2013

2.3 All examinations had a two o’clock start time; however a number of Providers had identified candidates as having special assessment arrangements which in some cases involved candidates commencing the examination earlier than the main cohort. In all such cases special arrangement candidates were not allowed to leave their assessment area until the commencement of the main cohort assessment, and secure delivery and collection arrangements were put in place for all examination materials.

2.4 A number of candidates attempted the assessments at locations overseas. The onus was placed on the Provider institution to ensure that a secure assessment centre was available in these cases, and the BSB required the start time of the examination at the overseas centre to be the same (or

4

similar to arrangements in 2.3) as the UK start time (regardless of time differences). To ensure the complete security of the examination papers the BSB dispatched all examinations to the overseas contact directly.

2.5 Provider institutions were given guidance on examination arrangements by the BSB. Exam invigilation reports were submitted by Providers to the BSB, detailing any major issues Providers believed may have had a bearing on the conduct of the examination itself at their assessment centres (for example, transport difficulties, bomb hoaxes, fire alarms, building noise).

2.6 Providers were also required to complete a pro-forma on each MCQ and SAQ question in each of the knowledge areas indicating their view as to the level of difficulty, relevance to the syllabus, whether a question was technically defective in some way, and the appropriateness of the solution. The pro-forma also provided an opportunity to comment on the paper as a whole. External examiners also submitted comments on the assessments.

CEB examiners were able to respond to the pro formas, as required, by issuing revised post-exam marking guidance in relation to the SAQ papers. All Provider comments were taken into account at the CEB meetings held to review MCQ and SAQ marks in each knowledge area.

3. The marking process

3.1 Candidates attempting the MCQ papers in each of the knowledge areas marked their answers on machine readable answer sheets. Provider institutions were required to take copies of the MCQ answer sheets and return the originals to the BSB. The MCQ scanning was undertaken by BSB support staff, using Speedwell scanners and software, who had undergone training provided by Speedwell.

3.2 SAQs were marked by staff at the Provider institutions following (revised) marking guidance provided by the CEB examiners. Provider marking teams are required to observe annotation conventions when marking SAQ scripts to show the impact of internal marking and moderation. Each Provider was required to ensure that an appropriately robust system of internal SAQ moderation was in place with particular regard being had to the pass/fail borderline candidates. The external examiner for the relevant knowledge area at each Provider was also required to sample SAQ papers as appropriate and require knowledge area teams to remark where it was felt guidelines had not been adhered to. For 2013 an enhanced sampling system was put in place whereby randomised samples in appropriate marking bands were selected by Providers for review by external Examiners, with copies of any scripts supplied to the External Examiners also being sent to the CEB examiners at the BSB. The range of scripts within the sampling bands was adjusted to reflect the fact that the effect of upward scaling by the CEB could result in candidates who, on first marking, appear to be clear fails in SAQ papers becoming borderline pass/fail candidates once the effect of scaling is applied.

3.3 Reports were provided by each Provider and the relevant BSB external examiner assigned to the Provider, commenting on the SAQ marking process in each of the knowledge areas, with a requirement to flag any issues that should be considered by the CEB in determining the marks to be scaled.

4. How the CEB reviewed the results

4.1 Two sub-boards were constituted to review the results in each of the three knowledge areas, one to look at MCQ results and one to look at SAQ results. Each sub-board was chaired by the CEB Chair, and attended by the relevant knowledge area examiners, an independent statistician and an independent observer.

4.2 At each sub-board the pre-scaling MCQ and SAQ results for 2011/12 and 2012/13 were noted, both the aggregate ‘all Provider’ passing rate (i.e. the percentage of candidates passing across all Providers combined), and the passing rates at each Provider.

5

4.3 Other key sources of information available to the sub-boards included:

Representations made by Providers in the MCQ/SAQ feedback pro-formas (see 3.3 above) and comments from CEB examiners.

A spread sheet showing the passing rate for each MCQ, and each component of each SAQ, achieved at each of the Providers cross-referenced to the representations made in the assessment pro-formas returned by the Providers (see 3.3 above) – thus flagging up any correlation of Provider criticisms and concerns with systematic poor performance by candidates.

• For the MCQs, ‘Manhattan’ diagrams indicating the extent to which each MCQ discriminated effectively between strong and weak candidates.

• For the SAQs, Provider feedback, BSB External Examiner comments, and any amended marking guidance issued by the CEB examiner teams.

• Evidence provided by routine random sampling any further requested sampling of SAQ scripts undertaken at the direction of the CEB

4.4 On the basis of the above evidence, and as advised by the independent statistician, the CEB sub-boards formulated a series of proposals, where required, for the scaling of cohort marks to be considered at the final CEB meeting. The CEB recognises that there are a multiplicity of approaches to cohort scaling and operates within the general principles set out in the scaling protocol adopted following consultation with the Providers. Amongst the options open to the CEB are:

• Crediting more than one correct answer to an MCQ • Disregarding an MCQ or part of an SAQ entirely if defective or inappropriate – no candidate

credited and maximum score recalculated • Crediting all candidates with the correct answer if an MCQ or part of an SAQ is defective or

inappropriate • Crediting all candidates with a mark or marks in respect of a part of an SAQ if there is

headroom to do so (i.e. provided a candidate has not already achieved the maximum marks available),

• Adjusting marks globally for an SAQ as a whole, an SAQ paper as a whole, or an MCQ paper as a whole to compensate for generic factors if there is headroom to do so (i.e. provided a candidate has not already achieved the maximum marks available),

• Adjusting marks for each candidate in a sub-cohort due to local assessment issues (not arising from the delivery of the course by a Provider, or matters related to the conduct of the assessment that can be dealt with through a Provider’s extenuation processes) provided the sub-cohort constitutes a statistically reliable sample for scaling purposes.

4.5 The final CEB Board meeting for the 2012/13 March/April examinations, held on 25th June 2013,

considered the recommendations of the sub-boards in respect of the knowledge area MCQ and SAQ cohort marks. The meeting was attended by the relevant examiners, key BSB officers, an independent statistician and an independent observer. The function of the final CEB board is to test the recommendations of the sub-boards, consider whether further sampling of SAQs may be necessary, and to confirm the MCQ/SAQ cohort marks subject to any outstanding QA issues related to on-going sampling. Once cohort marks are confirmed by the CEB they cannot subsequently be altered by Provider institutions. The process for challenging marks confirmed by the CEB is outlined in section 16 (below).

6

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Pas

sin

g ra

te %

MCQ Item Number

Professional Ethics MCQ pre-scaling 2012/13 first sit histogram question by question

5. 2012/13 first-sit results in Professional Ethics: MCQs

5.1 MCQ pre-scaling first sit passing rates – all Providers 2011/12 to 2012/13

Professional Ethics 2011/12 2012/13

MCQ pre-scaling all- Provider passing rate

92.6% 86.4%

5.2 MCQ pre-scaling 2012/13 first sit histogram question by question

This table indicates the range of pre-scaling passing rates for the Professional Ethics MCQs across all Providers, showing 1 MCQ with a pre-scaling passing rate of below 40%.

5.3 Issues arising at MCQ exam board

5.3.1 Shortly after the March 2013 Professional Ethics examination it was brought to the attention of the BSB that 6 questions in the MCQ section of the paper had originated from a mock assessment and solutions prepared for internal use by BPP during the 2010/11 academic year. Following investigations by the BSB it appears that the material in question had been offered to the BSB by BPP at the time material was being requested from Providers for the Question Bank. The material was supplied in good faith by BPP and inadvertently included in the main Question Bank used to generate centralised assessment. The questions were not tagged so as to alert CEB examiners compiling the paper that they had originated from a BPP mock paper. The questions derived from the BPP mock examination material were MCQs 2, 15, 16, 17, 18 and 19. BPP gave written assurances to the BSB that although the mock questions had been available to BPP students during 2010/11 via its intranet, the material had not been made available to BPP students in subsequent years and specifically had not been used in the preparation of students for assessment in 2012/13. The BSB received anecdotal evidence that some students had searched the internet for Professional Ethics revision help and had been able to locate the original 2010/11 BPP mock assessment and solutions prior to the 2012/13 first sit assessment.

5.3.2 The CEB reviewed the data in relation to the Professional Ethics MCQs to identify whether there was any statistical evidence to suggest an advantageous or disadvantageous impact as a result of this breach. It was confirmed that there was no statistical evidence of the results for the MCQs which

7

could point to an advantage or disadvantage to any cluster of students. The CEB confirmed that, on this basis, no intervention was justified in respect of the use of the BPP mock exam material.

5.4 Specific interventions and extent

Question Intervention applied CEB rationale

MCQ 6 Credit all students with 1 mark for MCQ 6

Error in the dates in the question

The CEB considered Provider comments received in relation to MCQ 6. The CEB Ethics Examiners confirmed that the dates within MCQ 6 were incorrect and therefore no possible correct answer was available to students within the options given. The independent statistician confirmed that the item discrimination of the question was good despite the erroneous dates. The CEB, following its practice in 2011/12 agreed that all candidates should be credited with having answered MCQ 6 correctly, including those who had not put any answer or multiple answers.

5.5 MCQ post-scaling first sit passing rates – all Providers 2010/11 to 2012/13

Professional Ethics 2010/11 2011/12 2012/13

MCQ post-scaling all-Provider passing rate

87.7% 92.6% 94.3%

The post-scaling MCQ first sit passing rate for 2012/13 of 94.3% shows the impact of the CEB intervention in raising the pre-scaling passing rate of 86.4% by 8%, in effect a 9.15 % increase as a proportion of the pre-scaling passing rate. The trend data for Professional Ethics MCQ passing rates over the three cycles of first sit assessments indicates a continuing slight increase in cohort performance with a 6.6% improvement in all-Provider MCQ post scaling passing rates since 2010/11.

5.6 MCQ pre and post-scaling first sit passing rates across all Providers 2012/13

The graph shows the effect of the intervention in respect of MCQ6 across all Providers. Providers are ranged in order of passing rates (high to low) according to their pre-scaling cohort performance. Hence Provider 1 had the highest pre-scaling passing rate but, following intervention, Provider 5 achieved the highest post-scaling passing rate, whilst the intervention had no apparent impact on passing rates at Provider 7. The effect of the intervention on the range of passing rates across Providers shows a compression from 8.4% pre scaling to 5.7% post-scaling.

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Professional Ethics MCQ pre and post scaling all Provider passing rates first sit 2012/13

MCQ 2012/13 pre scaling

MCQ 2012/13 post scaling

8

60

65

70

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Pas

sin

g ra

te %

Provider

Professional Ethics MCQ post-scaling first sit passing rates by Provider 2010/11 to 2012/13

% Pass MCQ Post Scaling2010/11



5.7 MCQ post-scaling first-sit passing rates by Provider 2010/11 to 2012/13

Providers are ranked 1 to 11 based on their 2010/11 post-scaling first sit Professional Ethics MCQ passing, showing how their 2011/12 and 2012/13 passing rates compare. Hence Provider 1 achieved a 100% passing rate in 2010/11 and has maintained a consistently high (95%+) cohort passing rate over the last two assessment cycles. It is noticeable that Providers 9, 10 and 11 have recovered from a low base in 2010/11 to produce consistently good cohort results. 5 out of 11 Providers are showing higher MCQ post-scaling passing rates since assessments were centralised.

6. 2012/13 first sit results in Professional Ethics: SAQs

6.1 SAQ pre-scaling first sit passing rates – all Providers 2011/12 to 2012/13

Professional Ethics 2011/12 2012/13

SAQ pre-scaling all-Provider passing rate

88.5% 86.7%

9

0

20

40

60

80

100

SAQ1aSAQ1bSAQ1c SAQ1 SAQ2aSAQ2bSAQ2c SAQ2 SAQ3aSAQ3bSAQ3cSAQ3d SAQ3

Pas

sin

g ra

te %

SAQ - sub-parts and totals

2012/13 first sit Professional Ethics all-Provider SAQ pre-scaling histogram question by question

(showing sub-parts)

6.2 SAQ pre-scaling first sit histogram question by question (showing sub-parts)

This table indicates the range of pre-scaling passing rates for the Professional Ethics SAQs across all Providers, clearly showing the relatively poor cohort performance in respect of SAQ2.

6.3 Issues arising at SAQ exam board

6.3.1 See para 5.3.1 (above) regarding the background to the issues arising in connection with the BPP 2010/11 mock assessment. The CEB noted that SAQ 2 on the 2012/13 first sit paper had been included in the 2010/11 BPP mock assessment, and considered the statistical data indicating that the passing rate for SAQ2 was considerably lower than the passing rate for the other two SAQs on the 2012/13 paper. The Chief Examiner for Professional Ethics advised the CEB that despite concerns relating to an unascertainable number of candidates having a “pre-existing familiarity” with the question and its answers, the raw statistical did not appear to indicate that such familiarity was widespread or, perhaps more intriguingly even if it was, that it stood any particular group of students in good stead. Performance was broadly poor on this question across the cohort. There was no evidence that any candidate could have known that SAQ2 would be used in the 2012/13 Professional Ethics first sit assessment, and there was no evidence to suggest that a significant number of candidates had derived any unfair advantage.

6.3.2 The CEB Professional Ethics examiners considered SAQ2 to be a good, if challenging, question, citing supporting evidence in this regard from an external examiner. It was acknowledged that the format for the question was not one previously employed in a centrally set Professional Ethics SAQ, nonetheless from the point of view of examiners sampling scripts, it appears not only to have been a good discriminator but also a question that really offered the chance to shine to those students who have begun developing the ‘ethical sense’ valued by the Professional Ethics examining team.

6.3.3 There were no other global or generic issues relating to the SAQ assessment.

10



SAQ 2 (c) Credit all students with 1 mark for 2 c) only where there is headroom to do so.

Acknowledgement of level of difficulty of the component

6.4.1 By way of intervention and scaling the CEB noted the criticisms of SAQ2(c), in particular the distribution of marks available in the marking scheme across the three sub-parts. In light of this issue, the overall advanced level of challenge in SAQ 2, the unexpected (from a student perspective) format of SAQ 2; and difficulty for students, even good ones, to achieve full marks on sub-part 2(c), it was agreed to award a universal single mark credit for this sub-part across the entire cohort, save for any student who had already achieved full marks on sub-part 2(c). The Independent Statistician and the Independent Observer confirmed that the SAQ2 issue had been considered in a proper, clear and transparent way by the Board.

6.5 SAQ post-scaling first-sit passing rates – all Providers 2010/11 to 2012/13


SAQ post-scaling all-Provider passing rate

90.2% 88.5% 89.5%

The post-scaling SAQ first sit passing rate for 2012/13 of 89.5% shows the impact of the CEB intervention in raising the pre-scaling passing rate of 86.7% by 2.79%, in effect a 3.22% increase as a proportion of the pre-scaling passing rate. The trend data for Professional Ethics SAQ passing rates over the three cycles of first sit assessments indicates almost no change in cohort performance since 2010/11.

6.6 SAQ pre and post-scaling first sit passing rates across all Providers 2012/13

The graph shows the effect of the intervention in respect of SAQ2(c) across all Providers. Providers are ranged in order of passing rates (high to low) pre-scaling. Hence Provider 1 had the highest pre-scaling and post-scaling passing rates. The intervention had very little effect in terms of changing the ranking order of Provider cohort performance, but clearly (as might be expected) had rather more

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Professional Ethics SAQ pre and post scaling all Provider passing rates first sit 2012/13

SAQ 2012/13 pre scaling

SAQ 2012/13 post scaling

11

70

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Pas

sin

g ra

te %

Provider

Professional Ethics SAQ post-scaling first sit passing rates by Provider 2010/11 to 2012/13

% Pass SAQ Post Scaling 2010/11 % Pass SAQ Post Scaling 2011/12

% Pass SAQPost Scaling 2012/13

impact on the relatively weaker cohorts in terms of raising their passing rates. The effect of the intervention on the range of passing rates across Providers shows a compression from 18.6% pre scaling to 15.2% post-scaling – significantly greater range than is the case with the corresponding MCQs.

6.7 SAQ first sit passing rate by Provider 2010/11 to 2012/13

Providers are ranked 1 to 11 based on their 2010/11 post-scaling first sit Professional Ethics SAQ passing, showing how their 2011/12 and 2012/13 passing rates compare. Hence Provider 1 achieved a 100% passing rate in 2010/11 but has seen a year-on-year decline since (albeit still a strong performance overall). Provider 6, by contrast, has consistently achieved top ranking since the centralised assessments were introduced in 2011/12. Similarly Provider 11 has recovered from a relatively poor performance in 20101/11. Providers 6, 9, and 11 recorded higher SAQ passing rates in 2012/13 compared to 2010/11.

12

6.8 MCQ and SAQ first sit post-scaling passing rates by Provider 2012/13 high to low

This table compares 2012/13 Professional Ethics MCQ and SAQ post-scaling passing rates across all Providers. The order of Providers is determined by the Provider’s combined post scaling passing rate for the MCQ and SAQs taken together (i.e. the percentage of candidates at the Provider passing both sections). What the data shows is that whilst the Provider 1 cohort did better in the SAQ assessment than the MCQ assessment, this pattern was not replicated across the other Providers, with the gap in passing rates between the two forms of assessment tending to widen where cohorts were weaker overall (with the exception of Provider 7). Indeed some of the weaker SAQ cohorts outperformed the better SAQ cohorts in respect of the MCQs, suggesting they were stronger at aspects of knowledge recall as compared to knowledge application.

7. 2012/13 first sit in Professional Ethics: combined MCQ and SAQ post-scaling results

7.1 Combined MCQ and SAQ all-Provider first sit post-scaling passing rates and grade boundary distribution 2012/13

Professional Ethics combined (MCQ & SAQ) all Providers post scaling first sit 2012/13

Total number sat - All Providers 1722

Pass Combined 1488 86.4%

Fail Combined 234 13.6%

Combined Grade Boundaries

Outstanding 349 20.3%

Very Competent 935 54.3%

Competent 204 11.8%

Not Competent 234 13.6%

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Comparing Provider cohort perfomance across Professional Ethics MCQs and SAQs post

scaling first sit 2012/13

MCQ 2012/13 post scaling


13

It should be noted that the CEB does not review, adjust or confirm the combined passing rate for any of the three knowledge areas. The combined results are simply the product of the relevant confirmed MCQ and SAQ marks which may or may not have been subject to adjustment by the CEB at an earlier stage. No further scaling is applied once MCQ and SAQ marks are combined to produce the passing rate for a knowledge area as a whole. Hence the overall knowledge area marks for cohorts and individual candidates, including grade boundaries, lie where they fall.

7.2 Combined MCQ and SAQ first sit pre and post-scaling passing rates by Provider high to low

The graph shows the combined effect of the interventions in respect of both the MCQs and SAQs across all Providers. Providers are ranged in order of pre-scaling passing rates (high to low). Hence Provider 1 had the highest pre-scaling and post-scaling passing rates. The intervention had very little effect in terms of changing the ranking order of Provider cohort performance, but clearly (as might be expected) had rather more impact on the relatively weaker cohorts in terms of raising their passing rates. The limited effect of the intervention on the range of passing rates across Providers shows a compression from 18% pre scaling to 15% post-scaling.

7.3 Combined MCQ and SAQ post-scaling first sit passing rates – all Providers 2010/11 to 2012/13


Combined MCQ and SAQ post-scaling all-Provider passing rate

85.5% 85% 86.4%

The post-scaling combined first sit passing rate for 2012/13 of 86.4% shows the impact of the CEB intervention in raising the pre-scaling passing rate of 82.11% by 4.3%, in effect a 5.23% increase as a proportion of the pre-scaling passing rate.

7.3.1 The data shows very little change in the all-Provider combined passing rates over the three first sit cycles to date.

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Professional Ethics Combined MCQ and SAQ pre and post scaling all Provider passing rates

first sit 2012/13

2012/13 pre scaling

2012/13 post scaling

14

60

65

70

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Professional Ethics combined MCQ and SAQ post-scaling first sit passing rates by Provider

2010/11 to 2012/13

% Pass 2010/11 % Pass 2011/12

% Pass 2012/13

0.00

20.00

40.00

60.00

80.00

100.00

1 2 3 4 5 6 7 8 9 10 11

% s

tud

en

t pe

r ca

tego

ry

Provider

Professional Ethics first sit 2012/13 post-scaling grade boundaries by Provider

Not Competent

Competent

Very Competent

Outstanding

7.4 First sit 2012/13 post-scaling grade boundaries by Provider

Providers are ranked according to their 2012/13 post-scaling combined Professional Ethics first sit passing rate. Hence Provider 1 has the highest post-scaling combined first sit passing rate and, predictably the lowest level of ‘Not Competent’ students. The level of ‘Not Competent’ students rises in line with the drop in the Provider passing rate. By contrast, the number of ‘Outstanding’ candidates presents a profile less closely linked to overall passing rates, hence the Provider ranked 5

th overall for

Professional Ethics achieves the highest level of ‘Outstanding’ candidates.

7.5 Combined MCQ and SAQ all-Provider first sit post-scaling passing rates trend analysis 2010/11 to 2012/13

15

Providers are ranked 1 to 11 based on their 2010/11 post-scaling first sit Professional Ethics SAQ passing, showing how their 2011/12 and 2012/13 passing rates compare. Hence Provider 1 achieved a 96% passing rate in 2010/11 but has seen a year-on-year decline since. Provider 3, however, has emerged as the strongest since the assessments were centralised in 2011/12. 5 out of 11 Providers are showing higher combined post-scaling passing rates since assessments were centralised.

7.6 All Provider first sit post-scaling grade boundaries trend analysis 2010/11 to 2012/13

Very little change in grade boundary distribution for Professional Ethics, but a slight increase in ‘Outstanding’ and ‘Very Competent’ for 2012/13 as against 2011/12.

8. 2012/13 first sit results in Criminal Litigation: MCQs


Criminal Litigation 2011/12 2012/13

MCQ pre-scaling all-Provider passing rate

86.8% 85.3%

0

10

20

30

40

50

60

Outstanding Very Competent Competent Not Competent

% s

tud

ents

Grade Category

Professional Ethics all- Provider first sit post-scaling grade boundaries trend analysis

2010/11 to 2012/13

2010/11

2011/12

2012/13

16

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

Pas

sin

g ra

te %

MCQ Item Number

Criminal Litigation MCQ pre-scaling 2012/13 first sit histogram question by question

8.2 MCQ pre-scaling first sit histogram question by question

This table indicates the range of pre-scaling passing rates for the Criminal Litigation MCQs across all Providers, showing 2 MCQs with a pre-scaling passing rate of below 40%, proportionately in line with the Professional Ethics MCQ pre-scaling outcomes.


8.3.1 The CEB meetings convened to consider the cohort results for the Criminal Litigation MCQs addressed the BPP 2010/11 mock assessment issues outlined at 5.3 (above). BSB staff reviewed the 2013 CEB First Sit Criminal Litigation examination papers in light of the BPP issue, and as a consequence of this investigation the Criminal Litigation paper had five MCQ questions reviewed/replaced before it was attempted by any candidates: MCQ13 – new question replaced duplication; MCQ 25 – amendment made to remove duplication; MCQ 32 – amendment made to remove duplication; MCQ 35 – amendment made to remove duplication; MCQ 36 – new question replaced duplication. The original examination papers were removed and destroyed and new examination papers were confirmed with the CEB examining teams in time for the usual distribution of the papers to Providers prior to the examination as scheduled. The BSB and CEB confirmed that the amendments made to the paper removed any repetition of the problems experienced with the Professional Ethics First Sit 2012/2013 examination paper.

8.4 There were no other global or generic issues relating to the MCQ assessment

17



MCQ 11 Credit answer [B] in addition to [C] Potential for more than one correct answer

MCQ 19 Credit answer [A] in addition to [B] Perceived ambiguity in question MCQ 20 Credit answer [A] in addition to [C] Potential for more than one correct

answer MCQ 23 Credit all students with 1 mark Perceived rule ambiguity

8.5.1 Due to pre-emptive action noted above at 8.3.1 the CEB confirmed that there was no requirement for intervention in relation to the BPP mock assessment issues.


Criminal Litigation 2010/11 2011/12 2012/13


91.8% 88.7% 88.9%

The post-scaling MCQ first sit passing rate for 2012/13 of 88.9% shows the impact of the CEB intervention in raising the pre-scaling passing rate of 85.27% by 3.67%, in effect a 4.3% increase as a proportion of the pre-scaling passing rate. The trend data for Criminal Litigation MCQ passing rates over the three cycles of first sit assessments indicates a broadly consistent pattern of all-Provider cohort performance within a 3% range.

8.7 MCQ pre and post-scaling first sit passing rates across all Providers 2012/13

The graph shows the effect of the interventions in respect of MCQs 11, 19, 20 & 23 across all Providers. Providers are ranged in order of passing rates (high to low) according to their 2012/13 first sit pre-scaling cohort performance. Hence Provider 1 had the highest passing rate both before and after CEB intervention. Generally the intervention favoured the weaker cohorts, particularly Provider 8, showing an uplift of over 7%. The effect of the intervention on the range of passing rates across Providers shows a compression from 18% pre scaling to 15% post-scaling.

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Criminal Litigation MCQ pre and post scaling all Provider passing rates first sit 2012/13

MCQ 2012/13 pre scaling MCQ 2012/13 post scaling

18

80

82

84

86

88

90

92

94

96

98

100

1 2 3 4 5 6 7 8 9 10 11

Pas

sin

g ra

te %

Provider

Criminal Litigation MCQ post-scaling first sit passing rate by Provider 2010/11 to 2012/13

2010/11 Pass Rate 2011/12 Pass Rate 2012/13 Pass Rate

8.8 MCQ post-scaling first sit passing rate by Provider 2010/11 to 2012/13

Providers are ranked 1 to 11 based on their 2010/11 post-scaling first sit Criminal litigation MCQ passing, showing how their 2011/12 and 2012/13 passing rates compare. Hence Provider 1 has maintained 1

st place ranking across three cycles of first sit assessments and has seen little movement

in terms of passing rates. Providers 5, 6 and 8 have stayed within a very narrow range close to their respective 2010/11 passing rates. Providers 9, 10, and 11 however have responded very positively to the challenges of centralised assessment and have consistently achieved passing rates ahead of 2010/11 levels.

9. 2012/13 first sit results in Criminal Litigation etc.: SAQs

9.1 SAQ pre- scaling all Provider first sit passing rate – 2011/12 & 2012/13

Criminal Litigation 2011/12 2012/13


73.9% 54.5%

Data for 2012/13 shows a significant decline in the pre-scaling all-Provider passing rate compared to 2011/12, but still fairly close to the pre-scaling position for Civil Litigation 2012/13

19

9.2 SAQ pre-scaling first sit histogram question by question (showing sub-parts)

This table indicates the range of pre-scaling passing rates for the Criminal Litigation SAQs across all Providers, clearly showing the relatively poor cohort performance in respect of SAQ2 and SAQ 5.


9.3.1 As detailed at 8.3.1 (above) the CEB considered the impact of the BPP 2010/11 mock assessment issue on the 2012/13 first sit Criminal Litigation SAQs and was able to confirm that none of the SAQs used in the 2012/13 paper were affected.

9.3.2 There were no other global or generic issues relating to the SAQ assessment



SAQ 2c Credit all students with 1 mark globally on SAQ paper, where there is headroom to do so.

Perceived level of difficulty and possible impact on the rest of the paper

SAQ 3c(ii) Credit all students with 1 mark for 3c)ii) only, where there is headroom to do so.

Possible drafting issue

SAQ 4d Credit all students with 1 mark for 4d) only, where there is headroom to do so

Possible drafting issue

SAQ 5a Credit all students with 3 marks for 5(a) only, where there is headroom to do so

Possible drafting issue and peripheral syllabus issue

9.4.1 The CEB noted the overall SAQ pre-scaling passing rate of 54.5% and noted the 30% difference with the MCQ passing rate prior to scaling (85.3%). The statistical results appeared to indicate that candidates had found the SAQ section of the examination significantly more challenging than the MCQ section. The CEB also noted that the pre-scaling SAQ passing rate for Criminal Litigation was comparable to the pre-scaling SAQ passing rate for Civil Litigation (52.1%).

0

20

40

60

80

100

SAQ

1a

SAQ

1b

SAQ

1c(i

)SA

Q1

c(ii)

SAQ

1c(i

ii)

SAQ

1

SAQ

2a

SAQ

2b

SAQ

2c

SAQ

2

SAQ

3a(i

)SA

Q3a

(ii)

SAQ

3b

SAQ

3c(

i)SA

Q3

c(ii)

SAQ

3d

SAQ

3

SAQ

4a

SAQ

4b

SAQ

4c

SAQ

4d

SAQ

4

SAQ

5a

SAQ

5bSA

Q5

cSA

Q5

d

SAQ

5

Pa

ssin

g ra

te %

SAQ - sub-parts and totals

2012/13 first sit Criminal Litigation all- Provider SAQ pre-scaling histogram question by

question (showing sub-parts)

20

9.4.2 The overall cohort performance appeared weaker than in the previous academic year; however, it was noted that two Providers had high pass rates and their performance was regarded as setting the benchmark for Provider cohort performance.

9.4.3 In relation to SAQ 2(c) in particular the CEB noted the EE’s and Providers’ comments and the fact that the two strongest Provider cohorts appeared to have struggled with the question. There was, therefore, some concern regarding SAQ 2(c) as an effective assessment instrument

9.5 SAQ post-scaling first sit passing rates all-Providers 2010/11 to 2012/13



86.2% 77.8% 69.9%

The post-scaling SAQ first sit passing rate for 2012/13 of 69.9% shows the impact of the CEB intervention in raising the pre-scaling passing rate of 54.5% by 15.47%, in effect a 28.42% increase as a proportion of the pre-scaling passing rate. The trend data for Criminal Litigation SAQ passing rates over the three cycles of first sit assessments indicates a year-on-year deterioration in cohort performance amounting to 16% across the three cycles to date.


The graph shows the effect of the four SAQ interventions detailed above at 9.4. Providers are ranged in order of passing rates (high to low) based on their pre-scaling cohort passing rates. Hence Provider 1 had the highest pre-scaling and post-scaling passing rates. The intervention had very little effect in terms of changing the ranking order of Provider cohort performance, but it is notable that Provider 4 saw its cohort passing rate rise by over 26%, compared with an increase of 2.4% for Provider 2. As might be expected, the interventions had rather more impact on the relatively weaker cohorts in terms of raising their passing rates. The effect of the intervention on the range (highest to lowest) of passing rates across Providers shows a compression from 46% pre scaling to 37% post-scaling – more than twice the range in passing rates for the corresponding MCQs.

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Criminal Litigation SAQ pre and post scaling all Provider passing rates first sit 2012/13

SAQ 2012/13 pre scaling SAQ 2012/13 post scaling

21

50

55

60

65

70

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Pas

sin

g ra

te %

Provider

Criminal Litigation SAQ post-scaling first sit passing rates by Provider 2010/11 to 2012/13

Pass rate % 2010/11 Pass rate % 2011/12 Pass rate % 2012/13


Providers are ranked 1 to 11 based on their 2010/11 post-scaling first sit Criminal Litigation SAQ passing rates, showing how their 2011/12 and 2012/13 passing rates compare. Hence Provider 1 achieved the top cohort passing rate in 2010/11 of 93% but has declined to below 70% since. By contrast Providers 6, 9 and 11 exceeded their 2010/11 passing rates in 2012/13.


50

55

60

65

70

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Comparing Provider cohort perfomance across Criminal Litigation MCQs and SAQs post scaling

first sit 2012/13

MCQ 2012/13 post scaling SAQ 2012/13 post scaling

22

This table compares 2012/13 Criminal Litigation MCQ and SAQ post-scaling passing rates across all Providers. The order of Providers is determined by the Provider’s combined post scaling passing rate for the MCQ and SAQs taken together (i.e. the percentage of candidates at the Provider passing both sections). The data shows Providers 1 to 3 with strong performances in both MCQs and SAQs, thereafter a marked divergence in performance with Providers 4 to 11 equalling or bettering Providers 2 and 3 in respect of the MCQs, but falling away dramatically in respect of the SAQs, again suggesting candidates at these Providers were stronger at aspects of knowledge recall as compared to knowledge application.

10. 2012/13 first sit post-scaling results in Criminal Litigation: combined MCQ and SAQ results

10.1 Combined MCQ and SAQ all Provider first sit post-scaling passing rates and grade boundary distribution

Criminal Litigation combined (MCQ & SAQ) all Providers post scaling first sit 2012/13





Outstanding 241 14%

Very Competent 736 43%

Competent 195 11%


It should be noted that the CEB does not review, adjust or confirm the combined passing rate for any of the three knowledge areas. The combined results are simply the product of the relevant confirmed MCQ and SAQ marks which may or may not have been subject to adjustment by the CEB at an earlier stage. No further scaling is applied once MCQ and SAQ marks are combined to produce the passing rates for a knowledge area as a whole. Hence the combined knowledge area marks for cohorts and individual candidates, including grade boundaries, lie where they fall.

10.2 Combined MCQ and SAQ first sit pre and post-scaling passing rates by Provider 2012/13

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Criminal Litigation combined MCQ and SAQ pre and post scaling all Provider passing rates first sit

2012/13 Combined 2012/13 pre scaling

Combined 2012/13 post scaling

23

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

% s

tud

en

t per

cat

ego

ry

Provider

Criminal Litigation first sit 2012/13 post-scaling grade boundaries by Provider

Not Competent

Competent

Very Competent

Outstanding

The graph shows the combined effect of the interventions in respect of both the MCQs and SAQs across all Providers. Providers are ranged in order of combined pre-scaling passing rates (high to low). Hence Provider 1 had both the highest pre-scaling and post-scaling passing rates. The intervention had a modest effect in terms of changing the ranking order of Provider cohort performance (Provider 11 moved up to 8

th for example), but the biggest impact was on Provider 5

where the passing rate increased by over 22%. The range of passing rates (high to low) narrowed from 45% pre-scaling to 31% post-scaling.




83% 74.7% 68.2%

The post-scaling combined first sit passing rate for 2012/13 of 68.2% shows the impact of the CEB intervention in raising the pre-scaling passing rate of 53.8% by 14.4%, in effect a 26.8% increase as a proportion of the pre-scaling passing rate. The trend data for Criminal Litigation combined passing rates over the three cycles of first sit assessments indicates a year-on-year deterioration in cohort performance amounting to 15% across the three cycles to date.


Providers are ranked according to their 2012/13 post-scaling first sit passing rates in Criminal Litigation. Hence Provider 1 has by some distance the highest percentage of ‘Outstanding’ candidates and lowest percentage of ‘Not Competent’. Provider 11, perhaps surprisingly, has the third highest percentage of ‘Outstanding’ candidates, whilst Provider 3 has less than 2%.

24

10.5 Combined MCQ and SAQ all-Provider first sit post-scaling passing rates trend analysis 2010/11 to 2012/13

This graph, which ranks Providers 1 to 11 based on their 2010/11 post-scaling first sit Criminal Litigation combined passing rates, shows how their 2011/12 and 2012/13 passing rates compare. Provider 1 dropped significantly between 2010/11 and 2011/12 and has seen a further, albeit less dramatic, decline in its passing rate for 2012/13. Provider 6 has maintained a steady level of performance across all three cycles, whilst Provider 9 has emerged as the strongest cohort. Providers 6, 9 and 11 show achieve passing rates above 2010/2011 levels, but only 2 Providers (2 and 8) improved on their 2011/12 passing rates.

10.6 All-Provider first sit post-scaling grade boundaries trend analysis 2010/11 to 2012/13

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Pas

sin

g ra

te %

Provider

Criminal Litigation combined MCQ and SAQ post-scaling first sit passing rates by Provider

2010/11 to 2012/13 % Pass rate 2010/11 % Pass rate 2011/12 % Pass rate 2012/13

0

10

20

30

40

50


% s

tud

ents

Grade Category

Criminal Litigation All Providers first sit post-scaling grade boundaries trend analysis

11/12/13

2010/11

2011/12

2012/13

25

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

Pas

sin

g ra

te %

MCQ item Number

Civil Litigation MCQ pre-scaling 2012/13 first sit histogram question by question

The three year trend analysis shows little movement in ‘Very Competent’ and ‘Competent’ grade boundaries, but a steady reduction in ‘Outstanding’ grades and a doubling of the percentage of candidates grades ‘Not Competent’.

11. 2012/13 first-sit results in Civil Litigation: MCQs


Civil Litigation 2011/12 2012/13

MCQ pre-scaling all-Provider passing rate

67.8% 46.3%

A marked decline (21%) in Civil Litigation MCQ pre-scaling all-Provider passing rates for 2012/13 compared with 2011/12, and lower than the corresponding passing rate for the Civil Litigation SAQ assessments.

11.2 MCQ pre-scaling first sit histogram question by question 2012/13

This table indicates the range of pre-scaling passing rates for the Civil Litigation MCQs across all Providers, showing 9 MCQs with a pre-scaling passing rate of below 40%, compared with 2 in the Criminal MCQ assessment.


11.3.1 The CEB meetings convened to consider the cohort results for the Civil Litigation MCQs addressed the BPP 2010/11 mock assessment issues outlined at 5.3 (above). BSB staff reviewed the 2013 CEB First Sit Civil Litigation examination papers in light of the BPP issue, and as a consequence of this investigation

26

the Civil Litigation paper had four MCQ questions reviewed/replaced before it was attempted by any candidates:

MCQ1 – new question replaced duplication;

MCQ 14 – amendment made to remove duplication;

MCQ 30 – amendment made to remove duplication;

MCQ 39 – new question replaced duplication.

11.3.2 The original examination papers were removed and destroyed and new examination papers were confirmed with the CEB examining teams in time for the usual distribution of the papers to Providers prior to the examination as scheduled. The BSB and CEB confirmed that the amendments made to the paper removed any repetition of the problems experienced with the Professional Ethics First Sit 2012/2013 examination paper.

11.3.3 Due to pre-emptive action noted above at 11.3.1 the CEB confirmed that there was no requirement for intervention in relation to the BPP mock assessment issues.

11.3.4 There were no other global or generic issues relating to the MCQ assessment


11.4.1 Following statistical analysis, overall passing rates and the comments received from Providers and External Examiners the CEB agreed the following interventions by way of scaling:


MCQ 4 Credit all students with 1 mark for MCQ4

Syllabus issue


Typographical error


Perceived level of difficulty and poor item discrimination

MCQ 17 Credit answer [B] in addition to [C] for MCQ 17

Potential for more than one correct answer


Syllabus issue


Perceived level of difficulty and poor item discrimination


Civil Litigation 2010/11 2011/12 2012/13


90% 78% 73.2%

The post-scaling MCQ first sit passing rate for 2012/13 of 73.2% shows the impact of the CEB intervention in raising the pre-scaling passing rate of 46.3% by 26.9%, in effect a 58.1% % increase as a proportion of the pre-scaling passing rate. The trend data for Civil Litigation MCQ passing rates over the three cycles of first sit assessments indicates a year-on-year reduction amounting to over 16%.

27

60

65

70

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Pas

sin

g ra

te %

Provider

Civil Litigation MCQ post-scaling first sit passing rate by Provider 2010/11 to 2012/13

% Pass Rate 10/11 % Pass Rate 11/12 % Pass Rate 12/13

11.6 MCQ pre and post-scaling first sit passing rates all Providers 2012/13

The graph shows the combined effect of the interventions in respect of MCQs 4, 9, 12, 17, 19, & 32 across all Providers. Providers are ranged in order of passing rates (high to low) according to their pre-scaling cohort performance. Hence Provider 1 had the highest pre and post scaling passing rate. Generally the intervention favoured the weaker cohorts, particularly Provider 10, showing an uplift of over 35%. The effect of the intervention on the range of passing rates across Providers shows a compression in the range of passing rates (high to low) from 45% pre scaling to 23% post-scaling. The biggest movement is in respect of Provider 7 which moves from 7

th rank pre-scaling to 4

th post

scaling.

11.7 MCQ post-scaling first sit passing rate by Provider 2010/11 to 2012/13

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Civil Litigation MCQ pre and post scaling all Provider passing rates first sit 2012/13

MCQ 2012/13 pre scaling MCQ 2012/13 post scaling

28

This graph ranks Providers 1 to 11 based on their 2010/11 post-scaling first sit Civil Litigation MCQ passing rates and shows how their 2011/12 and 2012/13 passing rates compare. Hence Provider 9 has emerged the strongest performer since the introduction of centralised assessments in 2011/12. Provider 1, the strongest cohort in 2010/11 has now become the weakest, based on 2012/13 cohort performance. Four Providers produced improved passing rates in 2012/13 compared with 2011/12 performances. Provider 9 was the only one to exceed its 2010/11 passing rate in 2012/13.

12. 2012/13 first sit results in Civil Litigation: SAQs

12.1 SAQ all Provider pre- scaling first sit passing rate 2011/12 and 2012/13

Civil Litigation 2011/12 2012/13


61.4% 52.1%

Civil Litigation all-Provider SAQ pre-scaling passing rate noticeably lower in 2012/13 compared with 2011/12, but still 6% higher than the corresponding Civil Litigation MCQ pre-scaling mark.

12.2 SAQ pre-scaling 2012/13 first sit histogram question by question (showing sub-parts)

It is clear from the above chart that SAQ1 was problematic in terms of cohort performance, in particular SAQ1(c). Candidates also performed poorly in respect of SAQ2(b).


12.3.1 As detailed at 11.3.1 (above) the CEB considered the impact of the BPP 2010/11 mock assessment issue on the 2012/13 first sit Civil Litigation SAQs and was able to confirm that none of the SAQs used in the 2012/13 paper were affected.

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

SAQ

1aSA

Q1

bSA

Q1c

SAQ

1

SAQ

2aSA

Q2b

SAQ

2c

SAQ

2

SAQ

3aSA

Q3

bSA

Q3

cSA

Q3

d

SAQ

3

SAQ

4a(

i)SA

Q 4

a(ii)

SAQ

4b

SAQ

4c

SAQ

4d

(i)

SAQ

4d

(ii)

SAQ

4

SAQ

5a

SAQ

5b

SAQ

5c

SAQ

5d

SAQ

5

2012/13 first sit Civil Litigation all-Provider SAQ pre-scaling histogram question by

question (showing sub-parts)

29

12.3.2 There were no other global or generic issues relating to the SAQ assessment


12.4.1 The CEB noted: (i) the all-Provider pre-scaling failure rate in relation to the Civil Litigation SAQs, (ii) the impressive performance of one candidate who had achieved a 98% mark for the Civil Litigation SAQs; (iii) that the pre-scaling SAQ all-Provider passing rate was higher than the corresponding passing rate for the Civil Litigation MCQs.

12.4.2 The comparison of results for 2012/13 with the previous year (2011/12) revealed that 4 Providers had achieved a pre-scaling Civil Litigation SAQ passing rate higher than their Civil Litigation SAQ pre-scaling passing rate for 2011/12.

12.4.3 The CEB acknowledged the difficulty of certain aspects of the SAQ paper and particularly in relation to the lack of clarity in SAQ1(c). In order to give maximum benefit to all candidates, the CEB agreed the that all candidates would be credited with 3 marks in respect of the SAQ paper as a whole, provided there was headroom to do so.

12.4.4. The CEB noted Provider comments in respect of SAQ2, in particular the wording regarding dates and time limits. The CEB also noted that some changes had been made in the post examination marking scheme which had been issued by the examining team for Civil Litigation to accommodate some of the points raised by the Providers. There were no other issues raised and no intervention was proposed.


SAQ 1c) Credit all students with 3 marks globally on SAQ paper, where there is headroom to do so

Perceived lack of clarity

12.5 SAQ post-scaling first sit passing rates – all-Providers 2010/11 to 2012/13



82.4% 73.5% 61.5%

The post-scaling SAQ first sit passing rate for 2012/13 of 61.5% shows the impact of the CEB intervention in raising the pre-scaling passing rate of 52.1% by 9.4%, in effect a 18.04% increase as a proportion of the pre-scaling passing rate. The trend data for Civil Litigation SAQ passing rates over the three cycles of first sit assessments indicates a year-on-year reduction amounting to over 21%.

30


The graph shows the effect of the SAQ 1(c) intervention detailed above at 12.4. Providers are ranged in order of passing rates (high to low) pre-scaling. Hence Provider 1 had both the highest pre-scaling and post-scaling SAQ passing rates. The intervention had very little effect in terms of changing the ranking order of Provider cohort performance, but it is notable that Providers 3 and 10 each saw a significant uplift in excess of 15%, whilst Provider 2 only improved its passing rate by just under 4%.Provider 10 in fact moves from 10

th place pre-scaling to 5

th post scaling. Scaling has limited

impact here on the range of passing rates across Providers. It is 40% pre scaling and 37% post-scaling.


35

45

55

65

75

85

95

1 2 3 4 5 6 7 8 9 10 11

Civil Litigation SAQ pre and post scaling all Provider passing rates first sit 2012/13

SAQ 2012/13 pre scaling


30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Pas

sin

g ra

te %

Provider

Civil Litigation SAQ post-scaling first sit passing rates by Provider 2010/11 to 2012/13

% Pass rate 10/11 % Pass Rate 11/12

% Pass rate 12/13

31

Providers are ranked 1 to 11 based on their 2010/11 post-scaling first sit Civil Litigation SAQ passing rates, showing how their 2011/12 and 2012/13 passing rates compare. Hence Provider 1 achieved the top cohort passing rate in 2010/11 of 93% but has declined to below 55% since. By contrast Providers 2,3,7 and 8 produced passing rates higher than their 2011/12 figures, and Provider 10 has achieved passing rates above its 2010/11 figures in both first sit cycles of centralised assessments.


This table compares 2012/13 Civil Litigation MCQ and SAQ post-scaling passing rates across all Providers. The order of Providers is determined by a Provider’s combined post scaling passing rate for the MCQ and SAQs taken together (i.e. the percentage of candidates at the Provider passing both sections). The data shows Provider 1 with strong performances in both MCQs and SAQs, Providers 1 to 4 with largely comparable MCQ and SAQ results, and Providers 5 to 11 with markedly different MCQ and SAQ scores, again suggesting they were stronger at aspects of knowledge recall as compared to knowledge application.

13. 2012/13 first sit post-scaling results in Civil Litigation etc.: combined MCQ and SAQ results

13.1 Combined MCQ and SAQ all Provider first sit post-scaling passing rates and grade boundary distribution 2012/13

Civil Litigation combined (MCQ & SAQ) all Providers post scaling first sit 2012/13





Outstanding 130 7.4%

Very Competent 563 31.8%

Competent 300 17%


50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Provider cohort perfomance across Civil Litigation MCQs and SAQs post scaling first sit

2012/13

MCQ 2012/13 post scaling SAQ 2012/13 post scaling

32

It should be noted that the CEB does not review, adjust or confirm the combined passing rate for any of the three knowledge areas. The combined results are simply the product of the relevant confirmed MCQ and SAQ marks which may or may not have been subject to adjustment by the CEB at an earlier stage. No further scaling is applied once MCQ and SAQ marks are combined to produce the passing rates for a knowledge area as a whole. Hence the combined knowledge area marks for cohorts and individual candidates, including grade boundaries, lie where they fall.

13.2 Combined MCQ and SAQ first sit pre and post-scaling passing rates by Provider 2012/13

The graph shows the combined effect of the interventions in respect of both the MCQs and SAQs across all Providers. Providers are ranged in order of pre-scaling combined passing rates (high to low). Hence Provider 1 had the highest pre-scaling and post-scaling passing rates. The intervention had a modest effect in terms of changing the ranking order of Provider cohort performance, but the biggest impact was on Provider 10 where the passing rate increased by over 27%. The range of passing rates (high to low) narrowed from 42% pre-scaling to 36% post-scaling.




79.5% 68% 56.2%

The post-scaling combined first sit passing rate for 2012/13 of 56.2% shows the impact of the CEB intervention in raising the pre-scaling passing rate of 37.1% by 19.1%, in effect a 51.5% increase as a proportion of the pre-scaling passing rate. The trend data for Civil Litigation combined passing rates over the three cycles of first sit assessments indicates a year-on-year reduction amounting to over 18%.

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Civil Litigation MCQ and SAQ combined pre and post scaling all Provider passing rates first sit

2012/13

Combined 2012/13 pre scaling

Combined 2012/13 post scaling

33


Providers are ranked according to their 2012/13 post-scaling first sit passing rates in Civil Litigation. Hence Provider 1 has the highest post-scaling first sit passing rate and predictably has the highest percentage of candidates graded ‘Outstanding’ and the lowest percentage graded ‘Not Competent”. The percentage of “Not Competent” gradings rises as the post-scaling first sit passing rates fall, whilst the distribution of “Outstanding” candidates is a little more uneven.

13.5 Combined MCQ and SAQ all Provider first sit post-scaling passing rates trend analysis 2010/11 to 2012/13

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11

% s

tud

en

tsp

er

cate

gory

Provider

Civil Litigation first sit 2012/13 post-scaling grade boundaries by Provider

Not Competent

Competent

Very Competent

Outstanding

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Pas

sin

g ra

te %

Provider

Civil Litigation combined MCQ and SAQ post-scaling first sit passing rate by Provider

2010/11 to 2012/13 Pass rate % 10/11 Pass rate % 11/12

Pass rate 12/13

34

This graph, which ranks Providers 1 to 11 based on their 2010/11 post-scaling first sit Civil Litigation combined passing rates, shows how their 2011/12 and 2012/13 passing rates compare. Provider 1 dropped significantly between 2010/11 and 2011/12 and has seen a further, albeit less dramatic, decline in its passing rate for 2012/13. Provider 10 has managed to improve on its 2010/11 passing rate in both years of centralised assessment, whilst only Provider 7 shows an improved passing rate comparing 2011/12 and 2012/13. Providers 2, 3 and 6 show little change between this first sit cycle and the previous one.

13.6 All Provider first sit post-scaling grade boundaries trend analysis 2010/11 to 2012/13

The three year trend analysis shows a noticeable drop in the percentage of candidates rated ‘Outstanding’, and a significant growth in the numbers rated ‘Not Competent’.

0

5

10

15

20

25

30

35

40

45

50


% s

tud

ents

Grade Category

Civil Litigation all-Provider first sit post-scaling grade boundaries trend analysis 2010/11 to

2012/13

2010/11

2011/12

2012/13

35

14 First sit modes of assessment and subject areas 2012/13

14.1 MCQ first sit post-scaling passing rates for 3 CEB areas 2012/13 compared

This graph compares the post-scaling first sit MCQ passing rates for the three knowledge areas across all Providers 2012/13. Providers are ranked according to the average of their post-scaling MCQ passing rates across the three knowledge areas. Hence Provider 1 had the highest average passing rate across the three MCQ assessments, but was in fact ranked second in the Professional Ethics MCQ assessment. Notwithstanding the differential level of passing rates for Criminal and Civil they track each other reasonably closely across the range of stronger to weaker Provider cohorts. The differential between the Criminal MCQ passing rate and the Civil MCQ passing rate is especially noticeable in respect of Providers 7 and 9.

65

70

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Knowledge area MCQ post scaling passing rates all knowledge areas all Providers first sit

2012/13

PE MCQ 2012/13 post scaling

Crime MCQ 2012/13 post scaling

Civil MCQ 2012/13 post scaling

36

14.2 SAQ first sit post-scaling passing rates for 3 CEB areas 2012/13 compared

This graph compares the post-scaling first sit SAQ passing rates for the three knowledge areas across all Providers 2012/13. Providers are ranked according to the average of their post-scaling SAQ passing rates across the three knowledge areas. Hence Provider 1 had the highest average passing rate across the three SAQ assessments. Provider 11, although coming last by some way in the rankings for the Criminal SAQs actually achieves 5

th place ranking for the Civil SAQs. Providers 4,6

and 11 have higher passing rates in Civil Litigation compared to Criminal Litigation.

50

55

60

65

70

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Knowledge area SAQ post scaling passing rates all knowledge areas all Providers first sit

2012/13

PE SAQ 2012/13 post scaling Crime SAQ 2012/13 post scaling

Civil SAQ 2012/13 post scaling

37

14.3 Combined MCQ and SAQ 2012/13 first sit post-scaling passing rates for 3 CEB areas 2012/13 compared

This graph compares the post-scaling first sit combined passing rates for the three knowledge areas across all Providers 2012/13. Providers are ranked according to the average of their post-scaling combined passing rates across the three knowledge areas. Hence Provider 1 had the highest average passing rate across the three combined assessments, with rates in a range between 95% for Professional Ethics and 80% for Civil Litigation. By contrast Provider 11 displayed a range of 87% for Professional Ethics down to 44% for Civil Litigation. Only Provider 6 achieved a higher passing rate in Civil Litigation compared with Criminal Litigation.

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Knowledge area combined MCQ and SAQ post scaling passing rates all knowledge areas all

Providers first sit 2012/13

Combined PE 2012/13 post scaling Combined Crime 2012/13 post scaling

Combined Civil 2012/13 post scaling

38

14.4 Comparison of Provider first sit post-scaling passing rates by type of assessment 2012/13

This graph compares the post-scaling first sit MCQ and SAQ passing rates for the three knowledge areas across all Providers 2012/13. Providers are ranked according to the average of their post-scaling combined passing rates across the three knowledge areas. The data shows a very strong performance from the Provider 1 cohort, achieving the 5 highest rankings (and one second place), with a range of only 10% between its highest and lowest passing rate. Provider 11 by contrast, apart from a creditable 5

th place ranking for Professional Ethics SAQs, was ranked in the bottom 3 for all

other assessments, with a range of over 42% between best and worst passing rate

50

55

60

65

70

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

MCQ and SAQ post scaling passing rates all knowledge areas all Providers first sit 2012/13

PE MCQ 2012/13 post scaling Crime MCQ 2012/13 post scaling

Civil MCQ 2012/13 post scaling PE SAQ 2012/13 post scaling

Crime SAQ 2012/13 post scaling Civil SAQ 2012/13 post scaling

39

14.5 Post-scaling 2012-13 first sit grade boundaries across the three knowledge areas compared

Professional Ethics Criminal Civil

Outstanding 20.3% 14.02% 7.4%

Very Competent 54.3% 42.81% 31.8%

Competent 11.8% 11.34% 17.0%

Not Competent 13.6% 31.82% 43.8%

The data shows a far higher level of ‘not competent’ candidates in the two litigation subjects.

15. First sit modes of assessment and subject areas – trends 2010/11 to 2012/13

15.1 Changes in all-Provider post-scaling first sit knowledge area subject passing rates 2010/11 to 2012/13

60

65

70

75

80

85

90

95

100

2010/11 2011/12 2012/13

Pas

sin

g ra

te %

Academic Year

All Providers First Sit post-scaling passing rate trends for SAQs and MCQs in all three knowledge areas for 2010/11,

2011/12, and 2012/13

Civil MCQ

Civil SAQ

Criminal MCQ

Criminal SAQ

Ethics MCQ

Ethics SAQ

40

The 2012/13 first sit assessments in the BPTC knowledge areas represent the 3rd

cycle of first sit assessments under the BPTC course (2

nd under the CEB processes). Hence it is now possible to look

across the three first sit assessment points to explore whether any discernible data trends are emerging. This table shows how passing rates have changed across both assessment modes in all three knowledge areas from the first sit in 2010/11 to the first sits in 2012/13. In 2010/11 the Providers set and marked the assessments. From 2011/12 the CEB determined the scope of syllabus coverage in the knowledge area assessments (through selection from the Question Bank) and it may be that broader syllabus coverage in Crime and Civil is proving challenging for some Providers and their students. Similarly, the use of CEB marker guidance and the random sampling of SAQ scripts by the CEB examiners may be reducing the freedom Providers had in the past to reward students for what they believed (rightly or wrongly) to be creditable answers to SAQs. The issue seems to be less acute in Professional Ethics where there is slightly more marker discretion and arguably less application of technical knowledge.

15.2 Change in Provider passing rates 2011/12 to 2012/13

-30

-25

-20

-15

-10

-5

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 11

% C

han

ge

Changes in Provider passing rates 2012 vs first sit 2013

Civil Litigation Change

Criminal Litigation Change

Professional Ethics Change

41

This shows a disappointing profile in terms of comparing Provider first sit passing rates for 2012/13 with 2011/12. Only 5 Providers are able to show some improvement over the previous year. Four show progress in Professional Ethics (where passing rates are already relatively high), two show progress in Criminal Litigation and only one in Civil Litigation. The year on year reversals in respect of Provider 2 and Provider 7 are particularly noticeable.

16. Post examination board issues

16.1 Administrative checks on data were conducted after the July 2013 final CEB meeting as a further quality assurance process.

16.2 BPTC candidates seeking confirmation of their marks are required to comply with the regulations and guidance available on the BSB website. In the first instance a candidate is required to seek a clerical error check from the relevant Provider. As a next step the candidate can seek an enhanced clerical error check from the BSB, for which a fee is payable. If any errors are found in the computation or transcription of marks, CEB Chair’s action can be taken to confirm the rectification with consequent endorsement at the relevant Provider institution.

16.3 The BSB has not put in place any process by which a candidate can ask for a remark of an SAQ paper, and there is no provision for any BSB appeal process that allows any challenge to the exercise of academic judgement on the part of the CEB. A candidate can submit a request for review of a CEB decision on the basis that the CEB has acted irrationally or unfairly in confirming the cohort marks for the centrally assessed examinations. An application for review must be made in the required form, within stipulated time limits and a fee is payable. The BSB will filter applications and refer those establishing an arguable case to the CEB Review Panel (an independent panel appointed by the BSB). If the Review Panel finds that there is evidence that warrants referral back to the CEB it will give reasons for its decision and invite the CEB to reconsider the decision in question. The guidance accompanying the regulations makes clear to candidates who have failed examinations and are required to take resits that an application for review should not be seen as an alternative to attempting the resit examinations, as any determination of the CEB Review panel and, in turn the CEB, is likely to be delivered after the time for resit examinations has elapsed.

17. Quality enhancement

17.1 The CEB regrets the issues that arose in respect of the 2010/1 BPP mock assessment material and its potential impact on the Professional Ethics assessment. Steps have been taken to ensure that the relevant material has been removed from the Question Bank, and draft assessments are specifically checked by BSB staff to check that no material from the BPP Mock assessment is included in any future CEB papers.

42

17.2 It was noted that a number of the CEB first sit assessments for 2012/13 contained typographical errors.

Assessment Question affected Nature of typographical error Professional Ethics MCQ 2 In the scenario ‘JEA Ltd’,

should have been ‘LEA Ltd’. Professional Ethics MCQ 6 Date error – ‘31 December

2015’, should have been 31 December 2014’

Professional Ethics MCQ 10 In the scenario ‘claim has been is listed’, should have been ‘claim has been listed’.

Professional Ethics SAQ 3 Auto correction in Microsoft word during printing, question labelling included ‘(a), (a), (c), (d)’, should have been ‘(a), (b), (c), (d).’

Civil Litigation, Evidence and Remedies

MCQ 9 In two answer options [B] and [D] there were two identical names of people, there should have been two different names.

The CEB reviewed all statistical evidence for these questions to ascertain whether the validity of the question had been undermined as a result of the typographical errors. To the extent that there may have been any impact on cohort performance appropriate adjustments have been made by the CEB as detailed in this report. The BSB and the CEB remain committed to ensuring the highest standards in the production of assessments and would seek to ensure, through a variety of paper scrutiny procedures, that such errors are either eradicated or kept to a minimum

17.3 The CEB is aware of a degree of Provider and student concern in respect of what some see as a disparity between the passing rates in Professional Ethics on the one hand, and the passing rates in the litigation subjects on the other, in particular Civil Litigation. Whilst the CEB does not accept that a disparity is necessarily a sign of something defective in respect of the assessment process, it is keen to understand the reasons behind the divergence of passing rates.

17.3.1 Two general points need to be made. The first is that results from 2010/11 are not necessarily the correct benchmark from which to judge passing rates achieved in 2011/12 and 2012/13. In 2010/11 Providers were setting and marking their own assessments. They may well have fashioned their assessments to reflect the emphasis of syllabus coverage in their classes. It would not be at all surprising to find passing rates falling following centralisation of assessment. The CEB examiners draft papers from across the entire published syllabus on the not unreasonable assumption that Providers moving to a centralised assessment system will have prepared their candidates accordingly. Significantly the CEB examiners found that some of the material supplied by Providers for the Question Bank in 2011/12 was not entirely fit for purpose, thus raising questions about the consistency of assessment across Providers prior to the introduction of the CEB. It should be recalled that one of the key aims of the Wood Report, in advocating the centralisation of assessment of the knowledge areas, was to improve standards and consistency in standards.

17.3.2 The second general point is that it may not be helpful to compare passing rates in Professional Ethics with the two litigation subjects. The subject matter in Professional Ethics is very different, the assessment allows for greater discretion in terms of what might be an acceptable answer, and it might be argued that there is a lower level of precise technical knowledge being applied.

17.3.4 It follows, therefore, that the more helpful point of comparison is between Criminal and Civil Litigation between 2011/12 and 2012/13 – the first 2 years of the centralised assessments. The data provided in this report shows that across these two first sit cycles there has been no change in the

43

Criminal Litigation MCQ passing rate; the Criminal litigation SAQ passing rate has declined by 7.7%; the Civil Litigation MCQ passing rate by 10.5%; and the Civil Litigation SAQ passing rate by 12%. Whilst these figures may cause concern for some, it should not be overlooked (as the data in this report shows) that certain cohorts are able to achieve very commendable passing rates in these assessments – suggesting that the assessments in question are effective in discriminating between weak and strong cohorts.

17.3.5 Although these matters are likely to be key issues in the forthcoming review of the first three years’ operation of the CEB, it may be worth noting that the extent of the syllabus in Civil litigation may also be a factor that is causing candidates to find the SAQ assessments more challenging. Syllabus coverage is an issue that is subject to a more immediate review in 2013/14; see 18.2 below.

18. Procedure and policy

18.1 Following consultation with the CEB Chief and Assistant Examiners for the knowledge areas it has been agreed that Providers will once again be required to supply additional MCQ and SAQ material that can be drawn on in compiling the 2014/15 assessments.

18.2 Representations have been sought from the Providers regarding the scope of the syllabus in each of the knowledge areas. A number of issues will be clarified in time for the start of the 2013/14 academic year. A more thorough review is on-going and the BPTC sub-committee will review the outcomes of these consultations, in conjunction with the CEB examiners, with a view to revised syllabus statements being in place for the 2014/15 academic year.

18.3 For the moment the CEB is continuing to recommend to the BSB that Providers are not identified when details of cohort performance in the centralised examinations are published. An immediate consideration is that there are 10 BPTC modules which represent 120 credits in total. Three of these modules are the centrally assessed knowledge areas, representing 25% of the course (30 credits). The remaining seven modules represent 75% of the BPTC final award (90 credits). It would be wrong, therefore, to imply that the performance of any given Provider in relation to the knowledge areas should be taken as a reliable proxy for the performance of that Provider in respect of the BPTC as a whole. The CEB preference is that identification of Providers and passing rates should only occur when there is a statistically reliable trend data (probably covering 3 to 4 years) that brings together Provider cohort performance in both the centrally assessed knowledge areas and the locally assessed skills areas.

Professor Mike Molan

Chair of the Central Examinations Board

25 July 2013

the bar standards board central examinations board · the bar standards board central examinations...

Documents