Keynote address Legal education research and evidence-based policy-making: ‘The nightmare and the noble dream’

Professor Julian WebbCanadian Association of Law Teachers

Annual Conference

Ottawa

2 June 2015

Structure

• Background – the LETR in England and Wales

• EBPM: ‘the nightmare and the noble dream’• How we sought to ‘do’ EBPM in an evidence-

lite environment• Implications of our experience

Context: lessons from the LETR

• Driver: Legal Services Act 2007 (E&W) –– ‘independent’

regulation and market liberalisation agenda

• Push from oversight regulator (LSB)…

David Edmonds – Lord Upjohn Lecture, November 2010

Announced regulators’ review in context of

concerns about quality of LET

Legal education and training is central to

encouraging ‘an independent, strong, diverse

and effective legal profession’.

“[Workforce development] is about achieving a constant

interplay between practice and education, with the two spheres

in constant dialogue, each driving improvement and

innovation in the other to the broader public good.”

The LETR process

• Stage 1: research-led; ‘cradle to grave’ – Phase 1: A literature review, including an element of

comparative work relating to other jurisdictions and professions;

– Phase 2: ‘Contextual analysis’ - identifying the knowledge, skills and attributes currently required in the legal services sector;

– Phase 3: ‘Workforce development’ - identifying potential future structural changes and their implications for future education and training needs

– Phase 4: Report and recommendations

• Stage 2: development and implementation by the regulators (BSB/IPS/BSB) - ongoing

The nightmare and the noble dream…

Evidence-based policy making

“What counts is what works”

Powell (1999: 23)

EBPM process (1): systematic review (Banks, 2010)

Evidence based

Policy

Transparency

Time

Methodology

Receptive policy environment

The nightmare? Legal education policy as a problem case for EBPM

• Lack of clear problem definition• Paucity of evidence• Political economy of legal education -

complex, contested terrain• ….W(h)ither EBPM?

Problem definition

• Emphasis on ‘big’ intangibles like quality, fitness for purpose, competence

• Complex assumptions of causality (and risks of PBEM!)

• Eg…

For example: assuring ‘competence’

Tearing up assumptions: in the real world...• Complexity of competence – soft skills, values, systems, etc• Level of qualification may be only weakly correlated to competence• Relationship between cost and quality of service is NOT linear

Initial and continuing competence• Inputs vs

outcomes and outputs

• Passive vs active approaches

• Balance between initial and continuing

• What would real assurance look like?

What about ‘Competence+’• Common standards +

a step beyond minimum regulation

• Re-accreditation?• Required PQ

specialisation ?

Quality and range of existing evidence

• We needed to construct the educational past/present in order to understand the future

• Legal education’s growing engagement with educational praxis, but

• Virtually no bibliographies• No meta-reviews• Very few systematic reviews

• And the quality of empirical research remains highly variable, with large gaps:

• Mostly small scale ‘one shot’ qualitative studies• Little proper (especially longitudinal) evaluation• Very little reliable statistical data• Little sense of historical emergence and continuity in the

research

Cochrane Collaboration

It’s the political economy, stupid!

• Law schools operating between the state, the university and the profession

• Reviews not driven by pedagogic issues: political engagements triggered by state or profession in response to perceived ‘problems’ - see further Webb (2016)

• Persistent demarcation disputes and seemingly irresolvable issues – have shaped successive reviews (eg ‘common training’)

• Tendency to create overlapping discursive and policy spaces between liberal and professional education and blur distinctions between them (cp LETR emphasis on “LSET’)

EBPM AND SOCIAL COMPLEXITY

EBPM: beyond the technical-rational paradigm?

EBPM assumptions (based on Banks, 2010)

Methodology

Necessary research skills and capacity

Resources to harvest and generate data

Note usually underlying cost-benefit model

Time

Good quality data is available

What about timeliness?

Transparency

Institutional independence (of advisors)

Right incentives (to deliver advice in public interest)

Structures and resources for public debate

Open data sets

‘New paradigm’: EBPM under conditions of social complexity

Lessons from complexity science, work on bounded rationality, ‘wicked problems’, systems theory, and modern regulatory theory….

(eg, Majone, 1989; Reed & Harvey, 1992; Sanderson, 2002, 2006; Baldwin & Black, 2008; Head, 2008, Wegner, 2009)

• Law of unintended consequences and the limits of regulatory steering

• Dampening effects on policy reform of internal system design/complexity

• Impact of complexity of social and governance networks

• One size does not fit all: possibly greater effectiveness of small-scale, participatory and localised interventions

• Beyond rational choice: values, ethical-moral choices, and desired ends matter

LSET as a ‘socially complex’ problem

Characteristics of socially complex problems

Corresponding features of LSET (eg)

There is no definitive definition of the problem

Some agreement over a need for reform, but widespread disagreement over the extent, priorities and nature of the changes required

They tend to be intractable General lack of effect from a number of recent education and training reviewsSpecific intractable problems: • Achieving consistency of standards• Reducing costs of training• Managing increasing numbers

The information needed to make sense of the problem is often ill-defined, changing and may be difficult to put into use

Currently operating in rapidly changing work and educational environments

Relative lack of robust, especially longitudinal, data

Costs of deriving meaningful information are relatively high

From Webb et al (2013), Table 1.1

Characteristics of socially complex problems

Corresponding features of LSET (eg)

They emerge in fields where there are multiple stakeholders; limited consensus as to who the legitimate stakeholders and/or problem-solvers are, and stakeholders are likely to have different criteria of success

Large number of stakeholders, with different understandings of the problem(s), and different levels of engagement with the process

Legitimacy questions exist, eg, over the extent of professional and regulatory interest in the Bachelor of Laws (LLB)

Evidence of different stakeholders having different ‘objectives’ for the review

Every attempt at a solution matters significantly

Reform tends to be a ‘one-shot’ operation so relatively high risk

Exacerbated by uncertainties about the new regulatory environment, and the tendency of LSET system to operate as a relatively low trust environment

Problem-solving under conditions of social complexity:

• Socially complex problems are more likely to require solutions that:– recognise that there are few right/wrong solutions as opposed to

better/worse outcomes;– build shared understanding of the problem amongst a range of

stakeholders;– build a shared commitment to action;– recognise that ‘one-shot’ reforms will have unintended as well as intended

consequences;– recognise that capacity for continuing engagement, and institutional

(re)design needs to be built-in to regulatory solutions.

• If solutions cannot be judged on right/wrong criteria then ‘social robustness’ / goodness of fit are viable grounds

1.18 an iterative approach that uses the methods of thematic inquiry (see Appendix D) to ground a process of collective learning and collaborative problem-solving. This three-stage process builds up a picture of the problem, including potential solutions to the problem, then identifies and addresses critical information gaps, before developing the actual solution(s) to the problem collaboratively with stakeholders

LETR methodology

• Triangulation – existing literature/data and LETR original research See Table 1.2 LETR Report Informed by both educational and regulatory theory

• Transparency Independent: regulators and stakeholders also research

subjects Open research process (so far as consistent with research

ethics)

• ‘Problem-based approach’ Iterative research method ‘Socially robust’ solutions

Problem-based approach (from LETR Report)

• LETR adopted an “an iterative approach that uses the methods of thematic inquiry to ground a process of collective learning and collaborative problem-solving. This three-stage process builds up a picture of the problem, including potential solutions to the problem, then identifies and addresses critical information gaps, before developing the actual solution(s) to the problem collaboratively with stakeholders.” (para 1.18)

• “An iterative approach was adopted which involved returning to issues as they became better defined. Because much of the data was to be based on perceptions and experiences it was necessary to triangulate information from different data sources so as to increase assurances as to the consistency and reliability of the findings.” (Appendix D, para 1.2)

Qualitative first and foremost

“Evaluation and assessment of LSET required evidence based upon the experience and judgement of stakeholders and participants in the provision and consumption of legal education and legal services. While some of these data were captured through questionnaire-based attitudinal surveys, qualitative methods were designed to discover meaning through fine attention to content, so interviews and focus groups allowed for a wider range of responses, with richer description and deeper analysis of the phenomena than would have been achieved by quantitative research.” (Appendix D, para 1.3)

Evidence base: www.letr.org.uk

• Thematic literature review (nearly 300 pages)• Qualitative research (focus groups/interviews; 306

respondents)• Quantitative surveys – ca. 1250 respondents• Re-run of 1991 ‘solicitors and their skills’ use of time

survey• Secondary (quantitative) analysis of Legal Services

Benchmarking Survey (consumer experience dataset)

• Stakeholder engagement (written responses to discussion papers, Symposium outputs, input and advice from ‘CSP’)

Consequences of approach

• Focus on ‘LSET’ system – primarily structures and processes, less emphasis on content

• Concern that regulation could act as a barrier to innovation and collaboration

• Need to create a ‘shared space’ for collaboration to happen

Main recommendations

• Outcomes-led system– ‘Day one’ outcomes defined by reference to the knowledge, skills and attributes of a

competent practitioner– Increase consistency in assessment standards– Greater co-ordination and collaboration between regulators in setting standards– Redefinition of content will follow; Report highlights concerns in respect of professional

ethics, communication skills, management skills, commercial awareness

• Structures– Focusing CPD on learning, not just “hours” – move to more outputs-driven model– Supervised practice (such as the training contract) no longer determined by time-served– Flexibility – more opportunities to work and study in different configurations– Access and diversity: Supporting apprenticeships, set standards for internships and work

placements, standards/qualification scheme for paralegals – Voluntary regulation of paralegals outside regulated organisations

• Better information– On diversity– On careers options, opportunities and risks– Greater coordination across system of information resources, research and development,

and evaluation

What sort of regulatory system did we envisage?

A version of OFR, risk-based, with the regulator as primus inter pares, part of the shared space, listening, encouraging, learning from other disciplines and professions, from empirical experiments and theory construction; and shaping the landscape of regulation to encourage innovation

(Paul Maharg in Ching et al, Upjohn Lecture, 2014)

Early responses…

• A missed opportunity• Long-winded(!)• Business as usual• Better than expected• A decent compromise• Sensible recommendations• An important milestone• Interesting times ahead

Reasons to be cheerful?

Hits?

• Outcomes relatively conservative

• No strong evidence that the system was not fit for purpose

• Progress:– on outcomes– Significant changes on CPD– Access and diversity

• Apprenticeships • Legal exec law firms• Voluntary regulation of

unregulated paralegals

Misses?

• Outcomes relatively conservative

• Over-dependence on ‘opinion’ evidence?

• Progress:– How common a framework?– Costs of training– Pace of change– Rejection of coordinating

‘Legal Education Council’

Implications

Some lessons to be learned

• POLICY-MAKING IS EXPERIMENTATION!

• Legal education reform – the never-ending story?

• Time and timeliness are critical

• The need for (moderately) big data – systemic failure so far

• Need for more sophisticated research/experimental design

• History repeats: importance of regulatory ‘mode’ and responsiveness

EBPM process (2): policy cycle: will it happen?

Systematic review

Incorporate into policy decisions

Oversee implementation

Monitor outcomes

Targeted evaluation

Implementation is everything

“ So whilst on the surface everyone (kind of) is saying they are (sort of) happy, no one is smiling. And no one is smiling because the research tells them there is not enough research; the research - and a moment's thought - tells them there is no easy solution; there are many hard yards; and it is everyone's problem, not yours, not mine, not theirs. It's ours.

The research stands as an astute reminder that this is the reality: implementation is everything. This will require resources (knowledge, skills, bodies, money) and it will need mechanisms for change.

For that reason (I surmise) they propose that the different constituencies coalesce around things like working out outcomes: what does a lawyer need to be able to do at day one. And they propose a Legal Education Council to advise regulators and act as focal point for sharing best practice. This kind of negotiated solution will require a great deal of energy and goodwill. I wonder if anyone is really up for it. A more likely scenario is the regulators who want change will liberalise, permit change and (if they do not skimp) properly assess it. That would require both a fleetness of foot and a robustness about testing the links between competence, education and training which we have yet to see attempted in the legal sector.”

Prof. Richard Moorhead, Legal Week blog, 27 June 2013

Developing a community of practice...

“A resounding sentiment was that there needs to be a permanent conversation about the future of the profession – from broad and creative imaginings of future business structures, to assessing emerging technologies, to pedagogical reform in the training of lawyers – to encourage lawyers to innovate within their own practice setting. More often than not, participants affirmed that there should be space within the CBA for these issues to be discussed on a more regular basis.” (Canadian Bar Association, 2014:24)

...the fundamental tension between education of lawyers as delivering public value and education of lawyers as delivering private value is structural. The tension may manifest itself in different ways under different conditions, but it will always be with us and must always be managed. Other matters likely to continually give rise to stresses, challenges, and the need for managing change are: the economics of law schools; the rapid evolution in the market for legal services; the function and value of accreditation standards; the financing of legal education; the role of parties other than law schools in legal education; and the role of media in understanding legal education and communicating with the public.

Since these forces and factors will always be with us, it is prudent for the system of legal education to institutionalize the process of dealing with them. All parties involved in legal education should support a framework for the continual assessment of strengths and weaknesses and of conditions affecting legal education, and for fostering continual improvement. The process should ensure that not only law schools, but also practicing lawyers, judges, and other interested actors have a voice and an opportunity for meaningful contribution.” (ABA Task Force, 2014:29)

Research capacity?

…there is also reason to question whether the thirty-year growth trajectory of socio legal scholarship can be sustained. The return of legal fundamentalism, epitomized by recent aggressive moves by the Federation of Law Societies of Canada to control law school curriculums and pedagogy, is likely to force them to divert their resources away from research towards professional training and to shift the focus of their faculty members towards publishing books and articles more explicitly designed for consumption by the practicing bar. The declining enthusiasm of governments for law reform, and social reform more generally, is also likely to trim socio-legal research budgets and to deprive socio-legal scholarship of the legitimacy it derived from its contributions to the reform of the legal system and of society.

• Harry Arthurs, The State of Canadian Socio-legal Research, CALT blog, http://www.acpd-calt.org/?page_id=2357

References

• Baldwin, R. & Black, J. (2008) ‘Really Responsive Regulation’ 71 Modern Law Review 59-94

• Head, B. (2008) ‘Wicked Problems in Public Policy’ 3(2) Public Policy 101-118• Majone, G. (1989) Evidence, Argument and Persuasion in the Policy Process (Yale UP)• Powell, M (ed) (1999) New Labour, New Welfare State? The Third Way in British Social

Policy (Policy Press)• Reed, M. & Harvey, D. (1992) ‘The New Science and the Old: Complexity and Realism in

the Social Sciences’ 22(4) Journal for the Theory of Social Behaviour 353-380• Sanderson, I. (2002) ‘Evaluation, Learning and Evidence-Based Policy Making’ 80(1) Public

Administration 1-22• Sanderson, I. (2006) ‘Complexity, “Practical Rationality” and Evidence-based Policy Making’

34(1) Policy and Politics 115-32 • Webb, J. (2016) ‘A Tale of Two Cities: Reflecting on Lord Neuberger’s “Reforming Legal

Education”’ in Duncan, Ashford & Guth (eds), Perspectives on Legal Education (Routledge, 2016)

• Webb, J, Ching, J, Maharg, P, & Sherr, A (2013) Setting Standards: The Future of Legal Services Education and Training in England and Wales (LETR Report) www.letr.org.uk

• Wegner, J, (2009) ‘Reframing Legal Education’s “Wicked Problems”’ 61(4) Rutgers Law Review 867

Sources on LETR

• Sherr, A and Thomson, S. (2013) ‘Tesco Law and Tesco Lawyers: Will Our Needs Change If the Market Develops?’ Oñati Socio-Legal Series, Vol. 3, No. 3, 2013. Available at SSRN: http://ssrn.com/abstract=2293660.

• Webb, J. (2013) ‘Regulating Lawyers in a Liberalized Legal Services Market: The Role of Education and Training’ 24(2) Stanford Law & Policy Review 533-70.

• Rt Hon Lord Neuberger of Abbotsbury, (2013) ‘Lord Upjohn Lecture 2012: Reforming Legal Education’ 47 Law Teacher 4-17

• Special Issue on LETR: Law Teacher, vol. 48(1) (2014).• H. Sommerlad et al (eds) (2015),The Futures of Legal Education and

the Legal Profession (Oxford: Hart) • J. Ching, P. Maharg, A. Sherr and J. Webb, (2015) ‘Lord Upjohn

Lecture 2014 - An Overture for Well-Tempered Regulators: Four Variations on a LETR Theme’ Law Teacher, vol. 49(1), forthcoming (eprint available online)