What are examiners thinking when they examine a thesis?

Margaret Kiley

CEDAM, The Australian National University

Margaret.kiley@anu.edu.au

Limits of this workshop…

• This is ‘just’ the theory• To get the practice in your own university you need to

contact your Graduate Studies authority• This workshop does not specifically focus on doctorates in

art, design and performance where an exhibition or equivalent plus exegesis is the norm

In pairs/small groups discuss

• One thing I don’t really understand about the examination process is…

• One of my biggest fears about the examination process is…

• Something I am hoping to learn from today’s session is…• Something I hope to contribute to today’s session is…

Preparing for Examination• What are your University’s expectations of a PhD or

Masters thesis? What is the difference?

• Who selects examiners? Does the candidate have a say?

• Can a candidate include published work? If so, how?

• Is there a word limit?

• What happens re multi-disciplinary research?

• How strategic can supervisors and candidates be with the selection of examiners e.g. is it a good idea to appoint an examiner who might be a future employer?

• What are the University’s rules on conflict of interest between candidate, examiner, and supervisor?

Overview

• Research re the selection of examiners • How examiners examine theses• Discussion

Selection of Examination (Kiley, 2009)

• Recent research suggests that the selection of examiners varies depending on the perceived quality of the candidate and the dissertation.

• Experienced supervisors report that there are two sorts of criteria that they use:

• Professional / academic• Personality attributes.

• The aim is to find examiners who are knowledgeable in the area and who will give a fair and balanced opinion.

Professional / academic considerations

• Topic/methodology fit• Understanding of the system: country, type of doctorate

(professional doctorate, exegesis)• Examiners outside the academy• Experience versus inexperience (avoid retirees with too

much time on their hands)• Availability

Personality Issues

• High standards but fair (avoid the pedants)• Intellectual courtesy and respect (avoid the Smart Alecs)• Reliability (avoid the ‘gunnas’)

Experienced Examiners report that they… (Mullins & Kiley, 2002)

• Expect the student to pass as they open the thesis• Are very reluctant to fail a student with most

experiencing considerable distress if they do so• Come to a decision about the quality of a PhD by about

the end of Chapter 2• Have a formative rather than summative view of thesis

examination• Believe that there is a risk attached to sending theses

to inexperienced examiners

Experienced Examiners appear to…

• Be fiercely independent in their views• Hold varying views about the purpose of the PhD. (Is it

the thesis or the student being examined?)• Consider professional duty as the main reason for

examining, followed by the fact that they are going to be needing examiners for their own students!

• Devote considerable time to examining each thesis • Have surprisingly inclusive approaches to

methodology/ paradigm• Demonstrate few discipline differences in their

responses, other than regarding publications

Inexperienced Examiners (Kiley & Mullins, 2004)

• Report a high level of confidence in their ability to examine (which is not always reflected in what they say in response to other questions)

• Suggest experience comes from supervising & examining Honours students and theses and reviewing manuscripts

• Adopt a similar approach to the process of examining as more experienced colleagues, although they are more likely to focus on the components of a PhD rather than the whole

Inexperienced Examiners …• At a surprisingly high rate, wanted to fail their first

thesis or said it was ‘awful’• Follow institutional criteria more than experienced

colleagues• Felt (some of them) they were being examined too• Suggest a main difficulty is inability to benchmark• See role as maintaining standards and performing

their summative assessment role ‘correctly’

Is there a difference?

• From work of Trafford (2003) from 130 vivas it was possible to determine that:

• Experienced examiners tended to ask questions that can be defined as ‘Defending doctorateness, contributing to knowledge, critique of research, synthesizing concept’

• Inexperienced examiners tended to ask more ‘technical’ questions

Trafford’s CategorisationInnovation and Development

HIGH

C. Questions generally related to issues such as research question, choice of topics, location of study

D. Defending doctorateness, contributing to knowledge, critique of research, synthesizing concept

A. Types of questions include resolving research problems, content, structure

B. Implications, awareness of, and familiarity with wider literature

Scho

lars

hip

&

Inte

rpre

tatio

n LO

W

LOW

HIG

H

Strategies for examining• There is no one way, but most:

• Begin by reading the Abstract, Acknowledgements, Introduction & Conclusion to gauge the scope of the work and whether what candidates say they are going to do is actually done

• Look at the references to see what sources have been used and whether they need to follow up on any of them, and to see if they have been referenced

• Read from cover to cover taking detailed notes, finally go back over the thesis to check whether their questions have been answered or whether their criticisms are justified

Tip

Strategies for examining

• There is no one way, but most:• Begin by reading the Abstract, Acknowledgements,

Introduction & Conclusion to gauge the scope of the work and whether what candidates say they are going to do is actually done

• Look at the references to see what sources have been used and whether they need to follow up on any of them, and to see if they have been referenced

• Read from cover to cover taking detailed notes, finally go back over the thesis to check whether their questions have been answered or whether their criticisms are justified

Analysis of reports demonstrates that… (Kiley, 2004)

A ‘good’ thesis has:• Critical analysis & argument• Confidence & a rigorous, self-

critical approach• A contribution to knowledge• Originality, creativity & a degree

of risk taking• Comprehensiveness &

scholarly approach• Sound presentation & structure• Sound methodology

A ‘less than ideal’ thesis has:•Too much detail with lack of

analysis•Lack of confidence, energy &

engagement by the candidate •Lack of argument and rigour•Shoddy presentation (typos

etc)•Lack of critique of own

analysis/ sweeping generalisations based on opinion rather than analysis

• Inadequate or poorly expressed methodology & scope

Failing a PhD is an awful experience….It can take months, and many re-readings, to finally come to that decision.

On the other hand…

• “It’s a delight to deal with a thesis which has a clear, concise, research question; a simple structure; a research design which was decided upon before the data was gathered; a literature review which relates to the topic; and a logical progression from ‘Results’ through ‘Discussion’ to ‘Conclusion’. These get marked briskly!”

Discussion and Questions

• In small groups list the main qualities of a ‘good’ doctoral thesis

• Through discussion then decide which are the top three qualities and why

• QUESTIONS?

ResourcesKiley, M. (2009). You don't want a smart alec: Selecting examiners of doctoral

dissertations Studies in Higher Education 34(8)Kiley, M. (2004). What examiners' comments can tell us about the postgraduate learning

environment. In C. Rust (Ed.), Improving student learning: Theory, research and scholarship (pp. 213-222). Hinckley, Leicestershire: The Oxford Centre for Staff and Learning Development.

Kiley, M., & Mullins, G. (2004). Examining the examiners: How inexperienced examiners approach the assessment of research theses. International Journal of Educational Research, 41(2), 121-135

Lees, B. (2007). Theses I have marked. Presentation at the Summer ACT Inter-University research workshop, ANU.

Mullins, G. & Kiley, M. (2002) ‘It’s a PhD, not a Nobel Prize’: How experienced examiners assess research theses. Studies in Higher Education 27(4) pp. 370-386

Research Supervision@ANU http://researchsuper.cedam.anu.edu.auSORTI web site at the University of Newcastle has information on examining theses,

especially in the performing/visual arts http://www.newcastle.edu.au/centre/sorti/publications.html

Trafford, V. (2003) Questions in doctoral vivas: Views from the inside, Quality Assurance in Education 11(2) pp 114-122