Student readiness Supervision challenges

New examiner challenges Challenges of format

Towards a ThesisAssessment Matrix: Anaction research project

Michelle Picard & Lalitha Velautham

Literature on thesis examination

• ‘There seems to be a tacit understanding of what constitutes

a well-prepared Ph.D. student’, and that ‘in the complete

absence any central repository of rules or a cosmic

accrediting agency’ there is ‘extraordinary stability’

(Kwiram, 2006 p. 142 cited in Holbrook et al, 2008).

• Experienced examiners generally give feedback

consistently. Inexperienced examiners find it difficult to

define acceptable thesis standards and are inconsistent in

giving feedback (Kiley & Mullins, 2004; Kiley et al, 2011).

• Need for indicators to distinguish between theses of

threshold quality and those of highest quality

(Holbrook et al, 2008)

• The global research/ the global thesis standard? RED, School of Education

Literature on assessment matrices

• Tools to negotiate expectations between

supervisors and students (Cadman & Cargill,

2007, Kumar & Strache, 2007)

• Taxonomies for reviewing literature (Cooper,

1988; Boote and Beile, 2005; Randolph, 2009)

• Thesis rating scale (Albertyn et al, 2011)

Areas for extension

• No clear empirical basis

• Lack descriptive language and quality indicators

commonly employed by experienced examiners

• Disciplinarily slanted towards Sciences

• Focussed on traditional thesis formats

Research aim

• To create thesis assessment discussion tools for

supervisors, students and (potentially) examiners

in order to clarify expectations and assist in

assessing research documents across formats

and disciplines

RED, School of Education

Theoretical Framework and Methods

RED, School of Education

Identification of issue

Development of theory

Development of matrices

Deployment & evaluation of matrices

(Drummond & Themessl-Huber 2007)

Action research spiral 1

1. Research proposal assessment matrix• Developed and refined based on discussions with

EAL students and their supervisors• Final product developed from RSD7 & Literature

Review (Willison, 2008 & Boote & Beile 2011)

• Used with over 600 research students from March 2010 and their supervisors as:

- Negotiation tool - Assessment tool • Overwhelmingly positive evaluation • Wordiness

RED, School of Education

Action research spiral 2

2. First draft thesis assessment matrix

• Imperatives from our experience & the literature as well as

work on Research Proposal Matrix

• Draft matrix developed from RSD7 & Literature Review (e.g.

Willison, 2009; Randolph, 2009; & Boote & Beile 2011) as well as insights

from spiral 1.

• Deployment to experienced supervisors at the

University of Adelaide via an online survey, by students in a

thesis writing group and at a postgraduate research

conference in South Africa (April, 2013).

RED, School of Education

Student comments Spiral 2

HDR students

• ‘It is good to have something to take in to discuss my writing with my

supervisor’ (HDR 15).

• ‘It helps me to know what I must do to do research in a PhD’

• ‘It is scary to see how much I still have to learn, I thought I knew all about

research in honours’ (HDR1

• ‘It is a relief to see I am ok, at a stage of development just like my

supervisor’ (HDR 21)

• ‘This grid really helped us to see what is expected of a thesis...you

know you say it is not a Nobel Prize, but sometimes my supervisors seem to

expect that... so it helps to know what a thesis really should do’.

• A bit ‘wordy’ (HDRs 1,12 & 16)

Conference comments

• ‘Too focussed on literature review and writing

skills’

• ‘Needs more focus on original contribution’

• ‘Good to consider thesis by publication’

• ‘Likely to be less useful in creative disciplines’

• ‘Does not really address argument-based thesis’

• ‘It still does not encapsulate the true work of the

PhD’

Experienced supervisors

Statements

1. Strongly agree 2. 3. 4. 5. Strongly disagree

A Thesis Assessment Tool could potentially 0help clarify thesis standards

with research students

16.7% (3)

41.2%* (7)

11.1% (2)

16.7% (3)

11.1% (2)

A Thesis Assessment Tool could potentially help new supervisors clarify

expectations of the thesis

16.7% (3)

52.9% (9)

0.0 % (0)

16.7% (3)

11.1% (2)

A Thesis Assessment Tool could potentially help new examiners clarify

thesis standards

29.4 % (5)

38.9 % (7)

0.0 % (0)

5.6 % (1)

22.2 % (4)

A Thesis Assessment Tool could potentially be useful to clarify standards

with international examiners

25.0 % (4)

31.3 % (5)

5.9 % (1)

18.8 % (3)

12.5 % (2)

A Thesis Assessment Tool could potentially be useful to clarify standards

in other thesis formats (e.g. by publication or exegesis)

11.1% (2)

35.3 % (6)

16.7% (3)

27.8 % (5)

5.6 %

(1)

A Thesis Assessment Tool could potentially differentiate clearly between

MPhil, PhD & Doctorate requirements

11.1% (2)

35.3 % (6)

33.3 % (6)

11.1% (2)

5.6 %

(1)

A Thesis Assessment Tool could potentially aid in researcher training

11.1% (2)

52.9 % (9)

5.6 %

(1)

11.1% (2)

16.7% (3)

Positive responses in….

• Clarify expectations for research students

• Clarifying expectations for new supervisors

• Clarifying expectations for new examiners (but

also high strongly disagree)

• Aid in researcher training

Positive comments

• ...this would be very helpful for examiners - provides clear differentiation and progression, and what is suitable for each level.(ID 12)

• ...any template which gives students, supervisors, etc. an understanding of levels of competency will assist them to distinguish between competent and excellent work. (ID 12)

• the criteria listed coincide with the long established measures of adequacy in our discipline. (ID 17)

Less favourable responses in…

• Assist new examiners (22.2 strongly disagree)

• Other thesis formats (27.8 disagree)

• Differentiate between MPhil, Doctorate, PhD (33.3 neutral)

Examiners & Supervisors

• ...it introduces thresholds that do not correspond with the Pass/Fail approach to research degree examination. There is absolutely no point in differentiating between 'Pass' and 'Excellence' at the PhD level. (ID 10)

• New students, supervisors and even examiners may still lack the knowledge or confidence, even with these criteria, to work out where their work sits... (ID 12)

• If a supervisor doesn't know what is entailed in a thesis of particular kind at a particular level, he/she should not be a supervisor. (ID 16)

Formats & disciplines

• Please note that creative theses (creative writing

PhD and MPhil) would need an adapted assessment

tool - to be developed by the relevant discipline(s).

One size does not fill all.(ID 3)

• fails to address metrics related to thesis writing

specifically in the Sciences related to areas of poor

experimental design, data analysis, and data

interpretation

Differentiation

• ‘It wasn't clear to me that the draft assessment tool provided these metrics, but rather was couching inadequacies within more general and rather heavily semantic terminologies - the RSD has its own lingua franca that may not be widely understood or appreciated, and certainly I'd suggest is beyond the interpretation of a busy international examiner. (ID 8)

• ‘Experienced examiners will dismiss it. Inexperienced examiners will be misled by it.’ (ID 10)

• ‘...it is implicitly language rich, and with 50% of our students coming to UA from o/seas, to introduce them to such refinement of language early in their research careers, is surely asking a bit much.(ID 8)

Differentiation

• If it is to have 'currency' it would need to have

greater clarity through simplicity. The essential feature

for a PhD, for example, is originality. This is the crucial

ingredient. For the MPhil, however, the originality looms less

large. (ID 10)

• ‘excellent research outcomes’ often overshadows ‘poor

thesis construction’. (ID 8)

• A PhD degree is examined on a Pass/Fail basis. There is no

recognised international concept of a 'Pass' PhD as opposed

to a 'Better than Pass' PhD. (ID 10)

Action research spiral 3

3. Refinement of thesis assessment matrix

- Audit of rubrics provided to Australian and

international thesis examiners by institutions

- An audit of recurring phrases indicating quality

and rigor in examiners reports

- Combination of all data into revised thesis

assessment matrix

- Thus far, audit of rubrics completed

RED, School of Education

Categories of phrases

Knowledge Contribution

Synthesis and Analysis of the Literature

Methodology Innovativeness and Critical Thinking

Results/Findings

Vocabulary/Presentation

Other Attributes

The expectation that a thesis displays new perspectives and/or advanced knowledge of research

The student shows familiarity with, and understanding of, the relevant literature

The techniques adopted are appropriate to the subject matter and are properly applied

Originality of approach orinterpretation

The results are suitably set out, and accompanied by adequate exposition

The thesis contains material worthy of publication in a form appropriate to the discipline

Research apprenticeship iscomplete and holder isadmitted to thecommunity of scholars inthe discipline

University of Adelaide 21

Rubrics Across GO8 Universities

• Knowledge contribution PhD

“Thesis as a whole is substantial and original contribution to

knowledge of the subject…” (ANU; UWA)

“Significant contribution to knowledge and understanding in the

relevant field” (MU)

“Makes a distinct contribution to knowledge” (UOS)

“Thesis displays new perspectives and/or advanced knowledge…”

(UNSW)

“makes a distinct contribution to knowledge because of the

originality of the approach and/or interpretation of the findings and

in some cases the discovery of new facts” (UM)

Rubrics Across GO8

• Knowledge contribution Masters

“Shows a sound knowledge of the subject…evidence of some

independence of thought” (UWA)

“Makes a contribution…” (MU)

“Appreciation of relationship of topic to wider field of knowledge”

(UOS)

“Displays new perspectives” (UNSW)

“Demonstrates advanced learning in research skills” (UM)

Potential matrix advantages

• Supervisors: Clear articulation of expectations and standards

• Examiners: Fair, transparent and consistent evaluation,

assistance in articulating concerns

• Students: Clear goals and direction leading to better self-

management techniques in thesis writing

RED, School of Education

A word of caution

• Hijacking by rampant managerialists/ administrators

• Difficulties of pinning down the ephemeral

• Danger of ‘marking’ rather than ‘examining’ (Ward, 2013).

• Matrix/grid implies grades – the PhD is not graded per se

• Disciplinary peer review element of becoming ‘part of the

club’, entering into an in-depth discussion could potentially

be lost

• Personally now with two PhD examinations – still need for

assessment and comments, but helped in examining

process

Next step in Action Research Cycle

• Audit of reoccurring phrases/ language used by

experienced examiners

• Refining current matrix (in handout)

• Trial of current matrix with examiner volunteers

Questions & Feedback

• Please contact:michelle.picard@adelaide.edu.au Or lalitha.velautham@adelaide.edu.au

RED, School of Education