the application of the solo taxonomy in evaluating the level
TRANSCRIPT
The application of the SOLO taxonomy in evaluating the level of introductory financial accounting assessments
G von Benecke
orcid.org/0000-0003-3480-8801
Dissertation accepted in fulfilment of the requirements for the degree Master of Commerce in Accountancy at the North-West
University
Supervisor: Prof S van Rooyen
Graduation: June 2021
Student number: 23227389
i
ACKNOWLEDGEMENTS
I would like to express my sincere gratitude and appreciation towards the following persons for
their support of this study:
My Heavenly Lord and Father, Jesus our Christ and Redeemer and the Holy Ghost. The
most merciful and Almighty. May I never forget that my purpose on this earth is to do all
for Your glory alone.
My wife Aletta, for her love, support and patience and the many cups of coffee along this
journey. I love you so much.
My broader family for their love, support and listening to many stories about SOLO even
when we were on holiday. I love all of you.
Prof Surika van Rooyen for her guidance, provision of reference material, patience and
above all the motivation she has given me. You were always friendly and reasonable. You
are a special person.
The universities that provided the data for this study. Thank you so much.
My NWU colleagues for their support and reference material.
The NWU bursaries department for funding my study.
The language editor, Mrs Ann-Lize Grewar, for her professional assistance.
ii
ABSTRACT
Key terms: Accounting education, assessment, constructive alignment, cognitive level,
professional accounting programmes, SOLO taxonomy
There is wide consensus that assessment drives student behaviour and that assessment needs
to be fair. In terms of a financial accounting module pitched at NQF level 5, universities
predominantly used the revised Bloom’s taxonomy to demonstrate an approximate cognitive
demand level for their summative assessments. This process remains largely subjective, since
the use of a verb (e.g. “apply”) to indicate a Bloom level is not particularly suited to financial
accounting, where one verb may imply different levels in different contexts. This pertinent issue,
together with the fact that a gap was found in the literature, leads to the primary objective of this
study, i.e. to provide a framework for the application of the SOLO taxonomy in evaluating the
standard of summative assessment of an introductory professional financial accounting module.
As part of achieving the study’s primary and secondary research objectives, a systematic
literature review was undertaken, which included contextualising summative assessment (dealing
with issues such as principles of assessment and stakeholders) as well as conceptualising the
application of the SOLO taxonomy as well as summarising and building on the work of other
scholars.
The chosen design and methodology of the study was driven by the research objectives as
underpinned by an overall philosophy of pragmatism. The nature of the research objectives and
data (being the financial accounting examinations) lead to a methodological choice of mixed
methods research, with the qualitative research design being dominant. The qualitative design
type is a document analysis specifically set in an interpretative paradigm mainly relying on the
technique of thematic analysis. The quantitative part of the study utilises content analysis and
basic descriptive statistics as part of the overall pragmatic approach to achieving the research
objectives.
The results of the study showed that the fundamental concepts of financial accounting (such as
recognition, measurement, etc.) as well as the applicable topics (as driven by the intended
learning outcomes and typically including assets, liabilities, income, expenses, etc.) can be used
as a proxy for cognitive demand induced by the assessment tasks. The combination of these
concepts and topics, together with the requirement (or lack thereof) of material relational activity
or extended abstract, can then be used to set a SOLO level for the assessment task. The SOLO
level of an assessment task may also be materially disrupted (or even set) by the themes of “time
allowed”, “layout” or “conventional wisdom”. A framework, which includes examples of
iii
assessment tasks as evaluated through the SOLO taxonomy, is given as the final product of the
research objectives being achieved.
iv
LIST OF ABBREVIATIONS
CAQDAS Computer-assisted qualitative data analysis software
CRA Criterion-referenced assessment
HEI Higher Education Institution
HEQSF Higher Education Qualifications Sub-Framework
IAESB International Accounting Education Standards Board
IES International [Accounting] Education Standard
IFAC International Federation of Accountants
IFRS International Financial Reporting Standards
ILO Indented learning outcome
NRA Norm-referenced assessment
NQF National Qualifications Framework
NQF Act National Qualifications Framework Act 67 of 2008
SAQA South African Qualifications Authority
SOLO Structure of the Observed Learning Outcome taxonomy
TLA Teaching and learning activities
USAf Universities South Africa (formerly known as Higher Education South Africa or
HESA)
v
TABLE OF CONTENTS
ACKNOWLEDGEMENTS ............................................................................................................. I
ABSTRACT .................................................................................................................................. II
LIST OF ABBREVIATIONS ....................................................................................................... IV
CHAPTER 1 INTRODUCTION ................................................................................................. 13
1.1 INTRODUCTION ................................................................................................. 13
1.1.1 Background ......................................................................................................... 13
1.1.2 Motivation of the study and previous research .................................................... 15
1.1.3 The context of the study ...................................................................................... 18
1.2 PROBLEM STATEMENT ................................................................................... 18
1.3 OBJECTIVES OF THE STUDY .......................................................................... 19
1.3.1 Primary objective/s .............................................................................................. 19
1.3.2 Secondary objectives .......................................................................................... 19
1.4 RESEARCH DESIGN AND METHODOLOGY ................................................... 20
1.4.1 Literature review .................................................................................................. 20
1.4.2 Empirical study .................................................................................................... 21
1.4.2.1 Target population and sampling frame ................................................................ 21
1.4.2.2 Sample method ................................................................................................... 22
1.4.2.3 Sample size ......................................................................................................... 22
1.4.3 Paradigmatic assumptions and perspectives ...................................................... 22
1.4.3.1 Ontological assumptions ..................................................................................... 22
1.4.3.2 Epistemological assumptions .............................................................................. 22
vi
1.4.3.3 Rhetorical assumptions ....................................................................................... 23
1.4.3.4 Methodological assumptions ............................................................................... 23
1.4.3.5 Theoretical framework ......................................................................................... 24
1.5 ETHICAL CONSIDERATIONS ........................................................................... 24
1.6 CHAPTER CLASSIFICATION ............................................................................ 25
1.7 SUMMARY .......................................................................................................... 26
CHAPTER 2 CONTEXTUALISING SUMMATIVE ASSESSMENT .......................................... 27
2.1 INTRODUCTION ................................................................................................. 27
2.2 CONSTRUCTIVE ALIGNMENT ......................................................................... 27
2.3 PRIMARY STAKEHOLDERS ............................................................................. 29
2.3.1 Students .............................................................................................................. 29
2.3.2 The Government of South Africa ......................................................................... 30
2.3.3 Professional bodies ............................................................................................. 31
2.3.4 Universities .......................................................................................................... 32
2.4 FUNDAMENTAL CONCEPTS OF ASSESSMENT ............................................ 32
2.4.1 Definitions of assessment ................................................................................... 33
2.4.2 Kinds of knowledge ............................................................................................. 34
2.4.3 Forms of assessment .......................................................................................... 35
2.4.4 Taxonomies ......................................................................................................... 36
2.4.5 Principles of credible assessment ....................................................................... 36
2.5 CONCEPTUALISING THE APPLICATION OF THE SOLO TAXONOMY IN
EVALUATING THE LEVEL OF SUMMATIVE ASSESSMENTS ....................... 38
2.5.1 Exploring the concept of SOLO ........................................................................... 38
vii
2.5.1.1 Origination of the SOLO taxonomy ..................................................................... 38
2.5.1.2 The meaning of the SOLO taxonomy .................................................................. 39
2.5.1.3 The SOLO taxonomy’s levels .............................................................................. 40
2.5.2 Review of other scholarly research on SOLO ..................................................... 44
2.5.2.1 Lucas and Mladenovic’s study ............................................................................ 44
2.5.2.2 Newton and Martin’s study .................................................................................. 45
2.5.2.3 Other studies ....................................................................................................... 46
2.6 SUMMARY .......................................................................................................... 47
CHAPTER 3 RESEARCH DESIGN AND METHODOLOGY .................................................... 48
3.1 INTRODUCTION ................................................................................................. 48
3.2 PHILOSOPHIES OF RESEARCH ...................................................................... 49
3.3 APPROACH TO THEORY DEVELOPMENT ..................................................... 55
3.4 RESEARCH DESIGN ......................................................................................... 58
3.4.1 Methodological choice ......................................................................................... 59
3.4.2 Research strategy ............................................................................................... 64
3.4.2.1 Case studies ....................................................................................................... 66
3.4.2.2 Document analysis .............................................................................................. 67
3.4.3 Time horizon ....................................................................................................... 70
3.4.4 Data collection and analysis (research methods) ............................................... 71
3.4.4.1 Target population and sampling frame ................................................................ 72
3.4.4.2 Sample method ................................................................................................... 73
3.4.4.3 Sample size ......................................................................................................... 74
viii
3.4.4.4 The process of coding ......................................................................................... 75
3.4.4.5 Thematic analysis ............................................................................................... 76
3.4.4.6 Content analysis .................................................................................................. 77
3.4.4.7 Textual analysis .................................................................................................. 77
3.4.4.8 Reliability and validity .......................................................................................... 78
3.5 SUMMARY .......................................................................................................... 80
CHAPTER 4 ANALYSIS AND FINDINGS ................................................................................ 81
4.1 INTRODUCTION ................................................................................................. 81
4.2 CONTENT ANALYSIS APPLIED IN SUPPORT OF THE THEMATIC
ANALYSIS .......................................................................................................... 81
4.3 ANALYSIS OF ASSESSMENT TASKS AND THE EMANATING THEMES ..... 87
4.3.1 Theme 1: Time allowed for the assessment tasks .............................................. 87
4.3.2 Theme 2: The layout of the assessment ............................................................. 91
4.3.3 Theme 3: Conventional wisdom .......................................................................... 94
4.3.4 Reflecting on Themes 1 to 3 ............................................................................... 97
4.3.5 Theme 4: The “dimensional method” of applying SOLO to the Financial
Accounting 1 examinations’ assessment tasks ................................................... 98
4.3.5.1 Example 1: Trial balance with adjustments ....................................................... 100
4.3.5.2 Example 2: Trial balance with adjustments and material integration of topics .. 101
4.3.5.3 Example 3: Disclosure of notes to the financial statements .............................. 102
4.3.5.4 Example 4: Unistructural question .................................................................... 103
4.3.5.5 Example 5: Theory and apply type question ..................................................... 103
4.3.5.6 Example 6: Cash flow statement ....................................................................... 105
ix
4.3.5.7 Example 7: Bank reconciliation ......................................................................... 106
4.3.5.8 Example 8: Financial ratios ............................................................................... 107
4.3.5.9 About journal entries and general ledger accounts ........................................... 107
4.3.6 Reflecting on Theme 4 ...................................................................................... 107
4.4 SUMMARY ........................................................................................................ 108
CHAPTER 5 CONCLUSIONS AND RECOMMENDATIONS ................................................. 109
5.1 INTRODUCTION ............................................................................................... 109
5.2 CONFIRMATION OF THE RESEARCH OBJECTIVES ACHIEVED................ 109
5.3 FRAMEWORK FOR THE APPLICATION OF THE SOLO TAXONOMY IN
EVALUATING THE LEVEL OF INTRODUCTORY FINANCIAL
ACCOUNTING ASSESSMENTS ...................................................................... 110
5.3.1 Introduction ....................................................................................................... 110
5.3.2 Time allowed to complete assessment tasks .................................................... 110
5.3.3 The layout of the assessment ........................................................................... 111
5.3.4 The effect of “conventional wisdom” ................................................................. 111
5.3.5 The link to a more structured and formalised way of applying SOLO to the
Financial Accounting 1 examinations’ assessment tasks ................................. 112
5.3.6 The “dimensional method” of applying SOLO to the Financial Accounting 1
examinations’ assessment tasks ....................................................................... 112
5.3.7 Conclusion ........................................................................................................ 126
5.4 LIMITATIONS OF THE STUDY ........................................................................ 126
5.5 AREAS FOR FURTHER RESEARCH .............................................................. 127
5.5.1 Topics noted as extending beyond the defined research objectives ................. 127
5.5.2 Further topics that may logically build on this study .......................................... 127
x
5.6 REFLECTIVE LEARNING ................................................................................ 128
5.7 SUMMARY ........................................................................................................ 128
BIBLIOGRAPHY ...................................................................................................................... 129
ANNEXURES ........................................................................................................................... 135
xi
LIST OF TABLES
Table 1-1: Examples of research involving Bloom's and the revised Bloom's
taxonomy ........................................................................................................ 15
Table 1-2: Examples of research involving the SOLO taxonomy .................................... 17
Table 3-1: The typical paradigms discussed by scholars ................................................ 51
Table 3-2: Further justification of the applicability of the interpretive paradigm ............... 52
Table 3-3: Assumptions under pragmatism ..................................................................... 55
Table 3-4: Typical characteristics of qualitative research as applied in this study ........... 61
Table 3-5: Reasons for using a mixed methods design ................................................... 64
Table 3-6: Typical techniques for analysing documents .................................................. 68
Table 3-7: Reflecting on the advantages and disadvantages of document analysis ....... 69
Table 3-8: Practical steps of document analysis ............................................................. 71
Table 4-1: An illustration of the word frequency in the examinations .............................. 85
Table 4-2: An illustration of the word frequency of the examinations’ assessment
tasks ............................................................................................................... 86
Table 4-3: Analysis of the time allowed for assessment tasks of the sample of
financial accounting exams ............................................................................ 90
Table 5-1: Research objectives achieved ...................................................................... 109
Table 5-2: Examples of typical Financial Accounting 1 assessment tasks with the
application of the dimensional method ......................................................... 115
xii
LIST OF FIGURES
Figure 2-1: External influences on, and alignment of, assessment .................................. 32
Figure 2-2: The “roots” of the SOLO taxonomy: Piaget’s four stages of cognitive
development of children's thought .................................................................. 39
Figure 2-3: A visual summary of SOLO ............................................................................ 44
Figure 3-1: The research “onion” ...................................................................................... 49
Figure 3-2: A schematic representation of deductive and inductive reasoning ................. 57
Figure 3-3: Sources and types of codes ........................................................................... 75
Figure 4-1: The “Word Cloud” illustrating the word frequency in the complete
examinations .................................................................................................. 85
Figure 4-2: The “Word Cloud” illustrating the word frequency in the examinations’
assessment tasks ........................................................................................... 86
Figure 4-3: The layout of Universities A to G’s examinations ........................................... 91
Figure 5-1: A visual summary of the SOLO taxonomy and assessment ........................ 114
13
CHAPTER 1 INTRODUCTION
1.1 INTRODUCTION
1.1.1 Background
There is wide consensus that assessment is a centre pillar of the learning process (Brown
2001:4). A major reason for assessment’s prominence is that it fundamentally affects students’
behaviour during the teaching and learning process (Rowntree 1987:1; Biggs & Tang,
2007:169). This also applies to financial accounting students in a South African context, since
assessment as a concept is key to measure whether these students demonstrate, as is
expected from them, a wide base of knowledge, skills, abilities and attributes by the time they
enter the accountancy profession (Van Rooyen, 2016:1). This agrees with the view of the
South African Qualifications Authority (SAQA), which states “… (a)s assessment is central to
the recognition of achievement, the quality of the assessment is therefore important to provide
credible certification.” (SAQA, 2001:16).
It is therefore important to recognise that amongst all the variables of the teaching and learning
process, assessment (and by analogy the standard thereof): i) facilitates student behaviour
such as deep learning (Biggs & Tang, 2007:25), and ii) has to incorporate the requirements
from various stakeholders such as the university, professional bodies and government policy.
An example of a stakeholder’s requirements is found in the government policy of the South
African Qualifications Authority’s (SAQA) level descriptors for the South African National
Qualifications Framework (NQF). The SAQA level descriptors are statements describing
learning achievement at a particular level of the NQF, and provides a broad indication of the
types of learning outcomes and assessment criteria that are appropriate for a qualification at
that level (SAQA, 2012:4).
Furthermore, important standard setting bodies such as SAQA (2016:1) and the International
Accounting Education Standards Board (IAESB, 2019:79) are in agreement as to the
importance of assessment as well as the principles underlying good assessment. These
principles are summarised as: fairness, validity, reliability, practicability, validity, equity,
transparency, and sufficiency within professional accounting education programmes. All these
principles can be collectively noted as “credible assessment” (SAQA, 2001:16).
Credibility of assessment is also recognised by Higher Education Institutions (HEIs) in South
Africa through their various assessment policies (NWU, 2011:1; CUT, 2016:3). An initial review
of some of these policies shows that in order to aid the implementation of principles of credible
14
assessment, documents are provided to evaluate the assessment tasks itself (NWU, 2011:1,
CUT, 2016:57). These documents are typically identified by descriptions such as
“Questionnaire on the Quality of Assessment Papers” (CUT, 2016:57), “Moderator's Report”
(NWU, 2012) or “Report by external examiner” (UNIZULU, 2015), etc. It is noted that these
reports almost universally utilise the revised Bloom’s taxonomy (refer below) to evaluate the
appropriateness of the cognitive demands (i.e. the level) of the assessment. This is no surprise
since a common conceptual framework is needed to assist in the alignment of the learning
outcomes with the assessment thereof (Lucas et al., 2014:578) and the best-known example
of such a framework, according to Herbert et al. (2009:23), is Bloom’s Taxonomy.
In 1956 Bloom’s Taxonomy was created as a classification system for educational goals
(Bloom et al., 1956; Bush et al., 2014:3). The major idea of the taxonomy is that what
educators want students to know, as encompassed in the statements of educational
objectives, can be arranged in a hierarchy from less to more complex (Huitt, 2011:1). The
taxonomy distinguishes six levels of cognitive processes (Clark, 2015), from knowledge (the
lowest level) to evaluation (the highest level) and it has been widely used to write objectives,
activities, and assessments (Bush et al., 2014:3). Bloom’s Taxonomy was later updated by
Anderson and Krathwohl (2001), who revised it to fit the more outcome-focused modern
education objectives (Huit, 2011:2). The revision included switching the names of the levels
from nouns to active verbs, reversing the order of the highest two levels and the addition of a
conceptualization of knowledge dimensions as factual, conceptual, procedural, and meta-
cognitive (Bush et al., 2014:2; Clark, 2015; Huit, 2011:2). This revision, which is known as the
revised Bloom’s taxonomy (Biggs & Tang, 2007:80), ranks the thinking skills in ascending
order as "remembering, understanding, applying, analysing, evaluating and creating" (Biggs
& Tang, 2007:81; Bush et al., 2014:2).
Biggs and Tang (2007:80) acknowledge that the revision of Bloom's Taxonomy is an
improvement, but that it still has some shortcomings such as that under the verbs of
“‘understanding’ you can find ‘identify’, ‘discuss’ and ‘explain’, which [can] represent three
different [cognitive] levels”. They therefore advocate their Structure of the Observed Learning
Outcome (SOLO) taxonomy as an alternative to the revised Bloom's taxonomy (Biggs & Tang,
2007:80). The SOLO taxonomy provides a systematic way of describing how a learner’s
performance grows in complexity when mastering many academic tasks (Biggs & Tang,
2007:76). It suggests five levels of understanding (in order of depth): pre-structural, uni-
structural, multistructural, relational and extended abstract (Biggs & Tang, 2007:79).
15
1.1.2 Motivation of the study and previous research
An initial review of the literature showed that some researchers typically used Bloom’s
Taxonomy and the revised Bloom’s Taxonomy to evaluate the cognitive level (demand) of
learning outcomes, student responses and/or assessment. Table 1-1 (Van Rooyen, 2016:95)
lists some examples:
Table 1-1: Examples of research involving Bloom's and the revised Bloom's taxonomy
Source: Van Rooyen (2016:95).
It is, however, due to the nature of a Financial Accounting 1 examination (especially in the
context of high stakes summative assessment), no easy task to evaluate this assessment to
determine a level of cognitive demand required, as explained in the following paragraphs.
Even with the aid of the revised Bloom’s taxonomy, this evaluation will remain a challenge.
Biggs and Tang (2007:80) support this observation by stating: “Anderson and Krathwohl’s
revision is an improvement, but even then under ‘understanding’ you can find ‘identify’,
‘discuss’ and ‘explain’, which represent three different SOLO levels”.
Authors Subject area Summary of findings
Example of studies using the original Bloom’s taxonomy
Momsen et al. (2010) Biology It was found that learning outcomes were written to
address the higher-order cognitive skills, but that
assessment tended to focus on the lower-order skills.
Yap et al. (2014) Accounting Results indicated that 80% of learning outcomes were
in the three lower cognitive levels (knowledge,
comprehension and application).
Ahmed et al. (2014) Engineering It was found that the cognitive domains were not
balanced within the degree programme.
Example of studies using the revised Bloom’s taxonomy
Williams (2013) Information
technology
The authors concluded that there was a shift towards
higher cognitive levels with progression through the
year levels of study. The revised Bloom’s Taxonomy
was found to be feasible in analysing the learning
outcomes for a course.
Lakshmi (2013) Finance The learning outcomes of finance modules were
categorised according to the revised taxonomy’s
cognitive levels. The lower-level skills along with the
application of knowledge dominated the learning
outcomes analysed. The results across the different
universities did, however, differ.
16
The major reason for this difficulty in evaluating a Financial Accounting 1 examination’s level
is that (in line with Biggs and Tang’s (2007:80) observation noted above), for an assessment
of Financial Accounting 1, verbs that indicate assessment tasks, such as “remember”, “apply”
or “evaluate”, which are usually associated with the revised Bloom’s taxonomy (Biggs & Tang,
2007:80-81) may imply different cognitive demands within a context which can differ
significantly for each assessment task, even if the same verb occurs.
Furthermore, Ramsden (cited by Biggs, 2007:xvi) states that
Biggs’s most outstanding single contribution to education has been the creation of the
Structure of the Observed Learning Outcome (SOLO) taxonomy. Rather than read about
the extraordinary practical utility of this device in secondary sources, get it from the
original here. From assessing clinical decision making by medical students to classifying
the outcomes of essays in history, SOLO remains the assessment apparatus of choice.
Literature (Table 1-2, page 17) on the SOLO taxonomy (defined in Biggs & Tang, 2007:76 and
discussed in paragraph 2.5.1.2 below) suggests it may well (amongst its various other uses)
provide a more objective and defensible way to evaluate a summative assessment task. A
relevant example (noted in Table 1-2, page 17) is found in Lucas and Mladenovic (2009:263),
where the application of SOLO in an introductory accounting course, regarding the evaluation
of the level of students’ understanding of subject specific concepts, is explored. None of these
scholars focused on the exact topic of this study, but their work can be incorporated and built
upon in order to meet the research objectives.
It is the application of the SOLO taxonomy to the summative assessment of a Financial
Accounting 1 module that is of interest to this study.
17
Table 1-2: Examples of research involving the SOLO taxonomy
Source: Researcher (adapted from Van Rooyen, 2016:95).
Authors Study’s title Subject area Method Scope Summary of findings Relevance
Newton and Martin
(2013:78)
Blooming, SOLO
Taxonomy, and
Phenomenography
as Assessment
Strategies in
Undergraduate
Science Education
Biochemistry Mixed (mainly
Quantitative)
Student responses of male
(n = 18) and female
(n = 10) students ages 20
to 36 (mean 24.5 ± 4.1)
enrolled in a second-year
bio-chemistry course in an
undergraduate Kinesiology
programme (2013:82).
Results suggest that
alternative assessment
strategies may be used as
tools to evaluate learning at
increasing levels of
cognitive complexity as
related to a deep learning
approach (2013:78).
The linking of the
phenomenography,
with SOLO
descriptors and
student responses
can provide
direction and
credibility to this
study.
Lucas and Mladenovic
(2009:263)
The identification of
variation in
students’
understandings of
disciplinary
concepts: The
application
of the SOLO
taxonomy within
introductory
accounting
Introductory
accounting
Qualitative Student responses on a
depreciation exercise were
obtained from 41 students
in their first semester of
introductory accounting at
a United Kingdom (UK)
university. Responses to a
‘profit and cash’ exercise
were obtained from 57
students in their second
semester of introductory
accounting in an Australian
university (2009:263).
Understandings evidenced
by the majority of students
may not meet the
expectations within Higher
Education. The findings
imply particular types of
interventions that could be
used immediately to support
changes in teaching and
learning approaches,
including assessment
(2009:277).
The linking of
SOLO descriptors,
student responses
and financial
accounting
concepts may
provide direction
and credibility to
this study.
18
1.1.3 The context of the study
Since this study contains a good measure of being cross-discipline (i.e. accounting and
education), it is useful to provide some context for the readers of the study.
It is therefore clarified that this study is concerned with the measuring and aiding of fair and
appropriate assessment for a student that is starting an accounting qualification (i.e. for
Financial Accounting 1) and that this qualification is primarily influenced by international
accounting standards (currently known as IFRS), professional bodies and government
regulations (e.g. SAQA). Although this study encompasses theory of education as a discipline,
the main aim and focus is on financial accounting as a professional discipline in a higher
education context.
It should also be mentioned that the term “introductory” in the study’s title refers to a financial
accounting module pitched at i) NQF level 5 in terms of ii) a financial accounting programme
(see paragraphs 1.4.2.1 and 2.3.3) that may typically lead to a professional designation. There
are other introductory accounting modules being part of, for example, a business course, of
which the goal is to give students an overview of different organisational functions. Such
introductory modules are not included in the scope of this study. A further comprehensive
discussion of stakeholders including SAQA and the NQF is given in paragraph 2.3 below.
1.2 PROBLEM STATEMENT
The above described background and motivation shows: i) the need for credible assessment,
as well as ii) the practical utility of the SOLO taxonomy in evaluating assessment tasks.
Although literature strongly supports these notions, there is a clear gap in the literature in that
the application of the SOLO taxonomy to evaluate the level of assessment of Financial
Accounting 1 written examinations (as a form of summative assessment) has not been
explored. As other studies (see Table 1-2, page 17) do show SOLO as a valuable tool of
analysis, it confirms that there is value in investigating the application of the SOLO taxonomy
in evaluating the standard of financial accounting assessments.
As part of the aforementioned, there is also the pertinent issue that, due to the nature of a
Financial Accounting 1 examination, it is no easy task to evaluate the assessment to determine
a level of cognitive demand required, even with the aid of the revised Bloom’s taxonomy. The
major reason for this is that for an assessment of Financial Accounting 1, verbs that indicate
assessment tasks, such as “remember”, “apply” or “evaluate” which are usually associated
with the revised Bloom’s taxonomy, may imply different cognitive demands within a context
that can differ significantly for each assessment task even if the same verb occurs.
19
The research problem can thus be summarised by the following questions:
How can the SOLO taxonomy be applied in order to evaluate the level of assessment
of a Financial Accounting 1 written summative assessment?
What are some typical themes and their influence on the standard or level of a
Financial Accounting 1 summative assessment that can be identified if the SOLO
taxonomy is applied as a tool for analysis?
1.3 OBJECTIVES OF THE STUDY
The following objectives have been formulated for the study:
1.3.1 Primary objective/s
The primary objective of this study is to provide a framework for the application of the SOLO
taxonomy in evaluating the standard of summative assessment of a Financial Accounting 1
module.
1.3.2 Secondary objectives
In order to achieve the primary objective, the following secondary objectives are formulated
for the study:
1. To contextualise from the literature the significant matters surrounding summative
assessment, including the matters of stakeholders’ interest and principles that should
be adhered to when setting and delivering assessments (Chapter 2).
2. To conceptualise the application of the SOLO taxonomy in evaluating the standard or
level of summative assessments (Chapter 2).
3. To document as part of the proposed framework, typical Financial Accounting 1
summative assessment tasks with the analysis through the SOLO taxonomy (Chapter
4).
4. To determine by analysis through the SOLO taxonomy, typical themes that have an
influence on the standard or level of a Financial Accounting 1 summative assessment
(Chapter 4).
5. To synthesise the insight gained through the study into a framework which can guide
the application of the SOLO taxonomy in evaluating the standard of summative
assessment of a Financial Accounting 1 module (Chapters 4 and 5).
20
1.4 RESEARCH DESIGN AND METHODOLOGY
In order to meet the research objectives, the study comprised a literature review and an
empirical study.
1.4.1 Literature review
There is little doubt that the literature review is of critical importance to every research project
(Mouton, 2013:86; Hart, 1998:13). It is in essence the review of the existing body of
scholarship and knowledge (Mouton, 2013:87) that can be built on with the main aim being to
add value and make a contribution to one’s field of research. It is evident from the research
objectives of this specific study that the literature review was compulsory.
The literature review has other important functions as well. The aims of the literature review,
as applied to this study, are as follows:
1.) “To ensure that one does not merely duplicate a previous study” (Mouton, 2013:87):
A review of the Nexus database that provides information on approximately 170 000
South African current and completed research projects including theses and
dissertations (NRF, 2015) was done. The North-West University’s electronic database
was searched (including, amongst others, EBSCOhost, WorldCat, Scopus, Lexis, Web
of Science, etc.) This review of Nexus and database searches showed that this study
is not a duplicate.
2.) Some of the aims can be grouped together as building on existing scholarship and
knowledge. Mouton (2013:87), for example, lists these aims as:
To discover the most recent and authoritative theorising about the subject;
To find what the most widely accepted empirical findings in the field of study are;
and
To ascertain what the most widely accepted definition of key concepts in the field
of study are.
The literature search provided references of scholarly work on assessment and the
SOLO taxonomy.
3.) The literature review sets the motivation for context and value of the research (Hart,
1998:14).
21
In order to achieve the above, the data sources for this study included relevant textbooks,
journal articles, unpublished studies, the NQF, the IES’s and SAQA’s guidance, and the
universities’ assessment forms.
The background section (paragraph 1.1 above) incorporates the result of the preliminary
literature review, which is expanded on in Chapter 2 below.
1.4.2 Empirical study
From the research objectives it can be seen that the Financial Accounting 1 summative
examinations, as a form of document that comprise mainly qualitative (textual) data, is
analysed by applying the SOLO taxonomy. Thus, qualitative research, using the document
analysis design as listed by Tesch (1990:58) and further outlined by Bowen (2009:27) and De
Vos et al. (2018:380), was mainly used for the empirical portion of the study (discussed in
paragraph 3.4.2.2). The main technique for analysing the documents was set as thematic
analysis including coding (paragraph 3.4.4).
The financial accounting examinations, however, also contain a limited amount of numeric
data (such as the number of marks), which is quantitative in nature. A limited analysis utilising
basic descriptive statistics (Saunders et al., 2016:527) as well as content analysis were used
with this data in order to support the qualitative approach. The study therefore ultimately drew
on mixed methods research as discussed by Creswell (2015:6) (see paragraph 3.4.1).
It can thus be seen that in the main, an inductive approach (Burney 2008:5) was followed by
transforming observations (the studying of the financial accounting exams) through analysis
(which included interpretation of the assessment task in terms of the SOLO taxonomy) into
more general themes. In line with Creswell (2014:186), a secondary deductive approach was
also followed, where the themes were considered against different exams in order to test their
value through applicability, but also to test whether there was further evidence that supported
the theme.
The empirical portion of this study also comprised the following methodology dimensions:
1.4.2.1 Target population and sampling frame
The target population, which is also the sampling frame for the study, is discussed and motived
in paragraph 3.4.4.1, and was set as final summative financial accounting examinations
pitched at NQF level 5 at the 26 public South African universities. In paragraph 3.4.4.1 it is
also motivated why 2016’s examinations are used but not detracting from the study’s validity.
22
1.4.2.2 Sample method
The sample method was motivated and set as non-probability purposive sampling (paragraph
3.4.4.2).
1.4.2.3 Sample size
The sample size consists of examinations from seven universities that were purposively
selected (paragraph 3.4.4.3).
1.4.3 Paradigmatic assumptions and perspectives
1.4.3.1 Ontological assumptions
Grey (2014:19) who builds on the work of amongst others Tesch (1994), Crotty (1998) and
Easterby-Smith (2002), describes ontology as the study of being, that is, the nature of
existence and what constitutes reality. Grey (2014:19) and Cohen et al. (2011:5) are in
agreement that in essence the researcher’s ontological stance is one of either i) reality being
external to the individual thus objective, positivist and nominalist in nature, or ii) there are
multiple realities and ways of accessing them and this is seen subjectively, a result of
individual cognition, relativist or nominalist.
In light of the research problem (paragraph 1.2) and objectives (paragraph 1.3), it can be seen
that this study is in essence one of describing “a way” of “how to”. This study is thus set in an
ontological stance of multiple and subjective realities since a solution will be proposed
(founded in research), but it is acknowledged that there can be other solutions and
interpretations. It is submitted that as long as the study is credible, this stance does not detract
from the value of the research.
1.4.3.2 Epistemological assumptions
While ontology embodies understanding what is, epistemology tries to understand what it
means to know. Epistemology provides a philosophical background for deciding what kinds of
knowledge are legitimate and adequate (Grey, 2014:19).
In line with the ontological stance motivated above, this study involves the researcher as the
main research participant as opposed to being an objective observer. In terms of Crotty (cited
by Grey, 2014:19) this can be stated as subjectivism. This epistemological stance is also
known as anti-positivism as is described by Cohen et al. (2011:6). In terms of this study, this
anti-positivist epistemology can be further refined as one of “interpretivism”. Grey (2014:23)
describes interpretivism as “the world is interpreted through the classification schemas of the
23
mind”. This description is apt since in terms of this study, documents will be analysed and
interpreted through the lens of the SOLO taxonomy.
Cohen et al.’s (2011:46) list of typical indications of applicability of the interpretative paradigm
is also applicable to this study. These include research that is qualitative, subjective, small
scale, interpretative, concerned with understanding and explanation, it concerns the individual,
is of practical interest, has personal involvement of the researcher, is non-statistical, and
accepts multiple directions of causality.
1.4.3.3 Rhetorical assumptions
According to O’Neil (cited by Young, 2013) rhetoric is the art of persuasion. Young (2013)
continues by explaining that the rhetorical structure of your paper is how you go about
persuading your reader that what you are saying is worth something, and that this structure
includes issues around the use of language and formatting. Mouton (2013:129-134) also
emphasises the importance of the style, content and form of scientific writing and provides 12
rules (that he compiled from various sources) to assist with effective scientific writing.
Young (2013) furthermore emphasises that the overall rhetorical assumption for qualitative
research is that it is not “truth seeking” or omniscient, but instead reporting what reality is
through the eyes of the research participants.
In the light of the above and in order to persuade the reader that this study is worthy of being
seriously considered, the study’s credibility will be enhanced through i) thick description of
logical and transparent reasoning where the reader will be able to follow the thought process
through transparent writing, as well as ii) the documenting of the systematic and referenced
process of this research in the research methodology chapter (Chapter 3).
1.4.3.4 Methodological assumptions
Methodological assumptions consist of the assumptions made by the researcher regarding
the methods used in the process of research (Dave, 2013).
As derived from Dave (2013), in terms of this study, the procedures used by the researcher
are mainly inductive and are based on the researcher’s own experience in collecting and
analysing data. The research here is the product of the values of the researcher. Through an
inductive approach, raw textual data is condensed into themes. Clear links are established
between research objectives and summary findings derived from raw data. A framework of
24
the underlying structure of experiences or processes that are evident from the raw data is then
developed. In Chapter 3 this study’s methodology (including assumptions) is comprehensively
described and motivated.
1.4.3.5 Theoretical framework
It can be seen throughout Biggs and Tang’s (2007) work that the SOLO taxonomy is
inextricably tied to, and flows from, their theory of constructive alignment (Biggs & Tang,
2007:50-62). The theory of "constructive alignment" entails the systematic alignment of the
intended learning outcomes, teaching and learning activities, and the assessment tasks while
knowledge is constructed throughout the process (Biggs & Tang, 2007:7). This specific theory
underpins this study and is further discussed in paragraph 2.2.
1.5 ETHICAL CONSIDERATIONS
As stated by Cohen et al. (2011:542), the overriding principle of ethical considerations is one
of “do no harm to participants”.
In light of this overriding principle the risk of any harm to participants was carefully considered.
This author, in light of his auditing and chartered accounting background, considers himself
capable of meticulously assessing risk.
The risk to participants of this study is subsequently considered minimal since:
1.) Data gathered (the financial accounting exams) is already in the public domain due to
the fact that students already wrote the exams and hence the exams can be obtained
from former or current students or from exam resources on different websites. An
example of easy access can be found on the website of Course Hero (2018).
2.) Initial discussions with possible participants (applicable representatives) indicated that
they are comfortable with especially the use of examinations that are at least one-year-
old (i.e. 2016 for this study) since the risk of being accused of “leaking” exams are then
completely avoided. The nature of Financial Accounting 1 topics, being about the
fundamental and introductory principles, leads to these exams still being sufficient for
this study’s purposes and hence this suggestion will be implemented.
3.) Although point 1 and 2 already show a low risk, the following principles of ethics, as
outlined by Smith (2011:99), will also be adhered to, so as to indicate how serious the
author considers ethics: Voluntary participation will be emphasised. Confidentially of
25
the research data will be strictly applied by electronic and physical protection of the
data. The researcher alone will have access to the primary data. The universities
whose exams will be analysed will be de-linked as specific identifiable participants from
the final research findings through the use of neutral labels.
Therefore, the nature of the study does not indicate any further risk of participation. In addition
ethical clearance was obtained for the study from the Faculty of Economic and Management
Sciences Research Ethics Committee (EMS-REC) (approval number NWU-00646-20-A4).
1.6 CHAPTER CLASSIFICATION
This study will comprise the following chapters:
Chapter 1 Introduction, background and planning
This chapter serves as an introduction to the study and provides an overview of the
background, motivation, research objectives and an outline description of the planned
research methodology of the study.
Chapter 2 Contextualising summative assessment
This chapter will be based on a systematic literature review which includes i) addressing the
research objectives of:
contextualising from literature the significant matters surrounding summative
assessment including the matters of stakeholders’ interest and principles that should
be adhered to when setting and delivering assessments (research objective 1);
conceptualising the application of the SOLO taxonomy in evaluating the standard or
level of summative assessments (research objective 2);
and ii) summarising and building on the work of selected other scholars.
Chapter 3 Research design and methodology
In this chapter the chosen design and methodology will be motivated and defended. This
includes paradigmatic assumptions.
Chapter 4 Analysis and findings
In this chapter the empirical evidence generated in order to further complete the following
research objectives will be documented:
26
As part of the proposed framework, typical Financial Accounting 1 summative assessment
tasks will be documented with an analysis through the SOLO taxonomy thereof (research
objective 3).
Typical themes that have an influence on the standard or level of a Financial Accounting
1 exam will be determined by analysis through the SOLO taxonomy (research objective
4).
The insight gained through the study will be synthesised to flow into the framework that
will be given in Chapter 5 and which can guide the application of the SOLO taxonomy in
evaluating the standard of summative assessment of a Financial Accounting 1 module
(research objective 5).
Chapter 5 Conclusions and recommendations
This chapter will include the study’s conclusions and recommendations. It will also clearly link
the findings to the problem statement and ensure that every objective has been achieved. The
final suggested coherent framework will be given. Areas for further research and reflective
learning will also be briefly discussed.
1.7 SUMMARY
This chapter served as an introduction to the study and provided an overview of the
background, motivation, research objectives and an outline description of the planned
research methodology of the study.
27
CHAPTER 2 CONTEXTUALISING SUMMATIVE ASSESSMENT
2.1 INTRODUCTION
This chapter will be based on a systematic literature review which includes i) addressing the
research objectives of:
contextualising from literature the significant matters surrounding summative
assessment including the matters of stakeholders’ interest and principles that should
be adhered to when setting and delivering assessments (research objective 1);
conceptualising the application of the SOLO taxonomy in evaluating the standard or
level of summative assessments (research objective 2);
and ii) summarising and building on the work of selected other scholars.
In paragraph 1.4.1 the purpose and function of the literature review was explained, while
paragraph 1.1.1 provided background to summative assessment and the importance of the
credibility thereof. In this chapter these matters will also be expanded on.
2.2 CONSTRUCTIVE ALIGNMENT
It is mentioned in paragraph 1.4.3.5 that the SOLO taxonomy is tied to the theory of
constructive alignment and that this theory also underpins the study. In the following
paragraphs the theory of constructive alignment is further discussed.
Biggs and Tang (2007:52) explain that the “constructive” part is based on the constructivist
theory, i.e. that learners use their own activity to construct their knowledge. Biggs and Tang
(2007:52) suggest that the best way to achieve this “construction of knowledge” is that the
learning outcomes (they refer to these as “intended learning outcomes” or “ILOs”) should
specify activities and content that students should engage in, while the teacher’s task is to:
i) set up a learning environment that encourages the student to perform these learning
activities, and ii) then to assess what the learning outcomes required in order to determine
whether the intended learning took place. Biggs and Tang (2007:13) refer to this process as
their version of the “often confused concept” of outcomes-based education, which they refer
to as outcomes-based teaching and learning (OBTL) which is “solely concerned with
enhancing teaching and learning” and which is to be implemented using constructive
alignment. Biggs and Tang (2007:21) acknowledge that this form of constructivist theory (and
28
their use thereof) has its roots in cognitive psychology as pioneered by Jean Piaget in 1950
(see Figure 2-2, page 39).
As is the case with any theory, this constructivist theory can of course be challenged, but what
is certainly applicable in this researcher’s experience is that deep authentic learning goes
hand-in-hand with true understanding. Understanding is seen as a personal process for each
individual. One cannot “understand” on someone’s behalf, that person must do it themself if
their desire is to learn the skill or content so as to be able to use it themself. A person wishing
to understand something must therefore construct a connection in his own mind. It therefore
seems logical and desirable that teaching and learning activities (TLAs) and assessment
should be geared towards assisting a learner in constructing connections in his mind (i.e.
“learning”).
Biggs and Tang (2007:52) furthermore explain that the “alignment” part of constructive
alignment means that learning activities, teaching activities and assessment tasks all stem
from well-written (in terms of using verbs) learning outcomes and therefore all having the same
aim.
Constructive alignment can thus be stated as the process where the intended learning
outcomes, the teaching and learning activities, as well as the assessment tasks are
systematically aligned, to be done in accordance with the learning activities required in the
outcomes (Biggs & Tang, 2007:7) so as to facilitate the construction of “understanding” and
ultimately true knowledge. One can also view this form of alignment as inherently constructive
(or “building-up”) in the same way as feedback or criticism is often said to be constructive.
Figure 2-1 (page 32) visually represents the process of constructive alignment. It can also be
seen that the assessment criteria have a pertinent place in this process.
In concluding this paragraph, it is submitted that it is hardly possible to overemphasise the
importance of constructive alignment in the current university environment (Brown, 2001:4).
In this environment, which includes increased student numbers, student-centeredness,
serious financial implications, diversity of students (Biggs & Tang, 2007:2), and various but
sometimes conflicting stakeholder interests (paragraph 2.2), it is vital to be able to
demonstrate fairness of assessment (paragraph 1.1.1) and authenticity of assessment
(paragraph 2.4.1) so as to be able to be accountable to all these different stakeholders whilst
coping with this ever-changing and demanding environment.
29
2.3 PRIMARY STAKEHOLDERS
The following paragraphs summarise, from the literature, context around the primary
stakeholders’ interest in summative assessment of introductory financial accounting modules
at public South African universities. The context is given per group of stakeholders.
2.3.1 Students
Brown (2001:6) lists three overall reasons for assessment at HEIs which are still applicable
today (the merits of these reasons are however beyond the scope of this study). These are: i)
to give a licence to proceed to the next stage or to graduation; ii) to classify the performance
of students in rank order; and iii) to improve students’ learning. These reasons also overlap,
but may unfortunately sometimes be in conflict (Brown, 2001:6). What is clear from these
reasons is that students are firstly and foremost the main stakeholders of assessment. Even
by reviewing just these three overall reasons for assessment, one can already see how
assessment may have a life-changing impact on students. Assessment can, amongst others
and especially in the South African context, literally mean the difference between a life of
poverty or a successful career. It is therefore no surprise that there is so much writing,
research, reflection and regulation of assessment worldwide.
Since students are the first primary stakeholders of assessment, it is apt to further discuss
their relationship with assessment. In this aspect it can be seen that scholars on assessment
are strongly in agreement that assessment heavily influences a student’s approach to the
learning process. This influence may in fact be so high that it can be seen that some authors,
such as Brown (2001:4), state that assessment actually drives student behaviour. Brown’s
(2001:6) first principle of effective assessment is that assessment shapes learning, so if you
want to change learning then change the assessment method. Biggs and Tang (2007:169)
agree by noting how teachers usually see the intended learning outcomes as the central pillar
in an aligned teaching system, but that students rather see assessment as the central pillar.
Ramsden (cited by Biggs & Tang, 2007:169) states that “From our students’ point of view,
assessment always defines the actual curriculum”.
Biggs and Tang (2007:169) explain that this effect that assessment has on student behaviour,
whereby assessment determines what and how students learn more than what the curriculum
does, is widely known as “backwash”. Biggs and Tang (2007:169) note that the term was
coined by Lewis Elton (1987). Biggs and Tang (2007:169) furthermore note that “backwash”
is usually seen in a negative light, especially in an exam dominated system, since students for
example spent their time in rote learning of prior year exam papers and focus on exam strategy
30
instead of substance of learning. All of this will lead to surface learning, a concept which
Biggs and Tang (2007:29) describe as learning activities that are too low a level as to achieve
the intended learning outcomes. An example of surface learning would be if rote memorisation
is used to falsely give the impression of real understanding (Biggs & Tang, 2007:29). In
contrast to surface learning, deep learning means activities where a student wants to, and
indeed does, develop real understanding and makes sense of what he or she is learning
(Biggs & Tang, 2007:24).
Biggs and Tang (2007:169) however, do explain that “backwash” can be utilised in a positive
way, mainly by two things being in place: i) constructive alignment (see paragraph 2.2) as well
as ii) criterion-referenced assessment (see paragraph 2.4.3). Doing this will harness students’
high concern for assessment and fit backwash into an overall process that facilitates the
desired effect of deep learning.
The above paragraphs focused on students as the main stakeholders when it comes to
assessment. It follows that students’ family, funders and prospective employers share this
stakeholder interest.
2.3.2 The Government of South Africa
In terms of Financial Accounting 1 as a module at a public South African university, the
government of South Africa is also a major stakeholder of university qualifications and their
eventual assessment processes (not least because it is a major funder of higher education).
It is therefore no surprise that these qualifications are regulated. The process of how such a
qualification is regulated will now be explained in the following paragraphs.
The South African Government is represented by the Department of Higher Education and
Training (SAQA, 2014:2) with its “mother” act being the Higher Education Act 101 of 1997, of
which the main goal was to establish a single co-ordinated higher education system.
Flowing from the mother act is one of the main “daughter” acts, which is the National
Qualifications Framework Act 67 of 2008 (the “NQF Act”), of which a main aim is to provide
for the National Qualifications Framework (NQF). Section 6 of this act mandates the NQF to
consist of ten levels including level descriptors (see SAQA, 2012).
In terms of section 10 of the NQF Act, a juristic person namely the South African Qualifications
Authority (SAQA), is enacted with its objective set as, amongst others, to oversee the further
development and implementation of the NQF. Sections 13-23 regulate SAQA’s further
functions.
31
The NQF Act, in terms of section 25, also enacts a Quality Council i.e. the Council on Higher
Education (CHE). Some of the primary functions of the CHE as prescribed by section 27 of
the NQF Act are i) the development, registration and publication of qualifications as well as ii)
to develop and implement a policy for quality assurance within its sub-framework.
The sub-framework referred to here (as enacted by section 7 of the NQF Act) is the Higher
Education Qualifications Sub-Framework (HEQSF) (Department of Higher Education and
Training, 2014:10) which build on the NQF’s levels to provide detail such as qualification
names, their purpose, characteristics and amount of learning credits.
The HEQSF (Department of Higher Education and Training, 2014:14) summarises the
interrelation of SAQA and the CHE’s roles as follows:
SAQA is responsible for the development of policy and criteria for registering standards
and qualifications on the NQF on the recommendation of the CHE.
The CHE is responsible for the development and management of the HEQSF and for
advising the Minister of Higher Education and Training on matters relating to the
HEQSF. This includes issues such as the development of naming conventions for
qualifications.
Sections 13 and 27 of the NQF Act require that both SAQA and the CHE should develop a
system of collaboration in terms of various issues, which include assessment.
The NQF Act in section 29, as well as the HEQSF (Department of Higher Education and
Training, 2014:14), also recognise the role of professional bodies. Section 29 of the NQF Act
broadly defines professional bodies as “statutory or non-statutory body of expert practitioners
in an occupational field” and provides for professional bodies to register with SAQA so as to
be recognised by the act. The HEQSF (Department of Higher Education and Training,
2014:14) provides for consultation with professional bodies.
2.3.3 Professional bodies
In terms of this study it is acknowledged that professional bodies have material influence on
the assessment in qualifications that eventually forms part of their requirements for
membership. The detailed requirements of the professional bodies are however beyond the
scope of this study. The target population of this study (paragraph 1.4.2.1) was set as
summative financial accounting examinations that are i) pitched at NQF level 5 and ii) forms
part of a financial accounting qualification. Such a qualification may (or may not) eventually
lead to a professional accounting designation. However, since the IAESB issued an
accounting education standard on assessment (see paragraph 2.3) and since this study’s topic
32
falls under accounting education (see paragraph 1.1.3), it made logical sense to discuss the
standard’s guidance. Furthermore, the IAESB (2019) as well as professional bodies such as
the South African Institute of Chartered Accountants (SAICA) are members of IFAC (2020).
2.3.4 Universities
Finally, it cannot be forgotten that the public universities themselves are primary stakeholders
of assessment; it is therefore stated in paragraph 3.4.4.1 that this study aims to firstly be of
benefit to the financial accounting programmes of South Africa’s public universities. South
Africa’s public universities’ membership organisation is Universities South Africa (USAf, 2020),
who amongst others, influences and contributes to policy positions regarding higher
education.
The section is now concluded with Figure 2-1, which is a visual generic summary of the
section.
Figure 2-1: External influences on, and alignment of, assessment
Source: Brown (2001:5)
2.4 FUNDAMENTAL CONCEPTS OF ASSESSMENT
This section deals with fundamental definitions and other core concepts around assessment
in general and summative assessment specifically. Recommended principles that should be
adhered to when setting and delivering credible assessments are included in the discussion.
33
2.4.1 Definitions of assessment
It is apt to begin the discussion with the concept of “assessment” itself. It is possible to provide
many definitions of assessment and it is also submitted that most readers intuitively
understand what “assessment” means, but as an anchor point the following definition is
provided. SAQA (2001:16) defines assessment as “A structured process for gathering
evidence and making judgments about an individual’s performance in relation to registered
national standards and qualifications”. The central idea here is one of having evidence
enabling measurement against a standard. Linking on to this notion of evidence is the concept
of assessment criteria. SAQA (2001:22) states that assessment criteria are “statements
whereby an assessor can judge whether the evidence provided by a learner is sufficient to
demonstrate competent performance”. Assessment criteria are clear, precise and transparent
statements against which performance is assessed (Geyser, 2004:95). Biggs and Tang
(2007:215) mention the use of rubrics as a way to set assessment criteria for some
assessment tasks. The use of assessment criteria fit into the overall process of constructive
alignment (and ultimately fairness of assessment), as discussed in paragraph 2.4.1 below.
Since the title of the study makes it clear that summative assessment is of interest, it is apt
to discuss what is meant by “summative assessment”. The concept of summative assessment
relates to the purpose of the assessment and is often contrasted to assessment with a
formative purpose (formative assessment) (Van Tonder, 2010:19). Hence, if the purpose of
the assessment is to support and improve the learning process, then the assessment is
formative in nature and known as formative assessment, whereas if the purpose of an
assessment is to assess the knowledge, skills, attitudes and/or values of a student at the end
of a particular period of learning, it is known as summative assessment (Van Tonder, 2010:19).
Biggs and Tang (2007:97) agree with these definitions and make it clear that “(f)ormative
assessment is provided during learning, telling students how well they are doing and what
might need improving; summative after learning, informing how well students have learned
what they were supposed to have learned”.
Biggs and Tang (2007:163) also refer to formative assessment as “formative feedback” and
summative assessment as “summative grading” to emphasise that the purpose of these two
forms of assessment is very different. One may wonder why this study then focusses on
summative assessment. The answer is that summative assessment is traditionally used at
public South African universities to grade students at the end of a module or year of study and
to eventually award a qualification or not (SAQA, 2014:8). This is also the case in Biggs and
Tang’s (2007:164) experience where they state that “(i)n summative assessment, the results
are used to grade students at the end of a course or to accredit at the end of a programme”.
34
It can thus be seen that summative assessment is “high stakes” and has serious financial and
other implications for a student (this however does not mean formative assessment is not
important). It is therefore due to this inherent serious implication of summative assessment
crucial that it is as fair as possible. One way of enhancing or demonstrating fairness is by
using a taxonomy such as Bloom’s or SOLO, to more objectively measure the level (or
standard) of a summative assessment.
It was further noted that the financial accounting examinations that make up the data set of
this study is a form of traditional or conventional assessment, since these written
examinations have been used over a long period (Van Tonder, 2010:19). Alternative
assessment utilises instruments that have not been used in particular discipline (e.g. financial
accountancy) for a long time and can be seen as relatively new or innovative, but they are
admittedly not necessarily authentic in nature (Van Tonder, 2010:19). Authentic assessment
is described by Van Tonder (2010:20) as assessment performed in contexts very closely
related to real life situations. Examples of alternative assessment instruments (which is the
case in terms of financial accountancy) include presentations by students, reflective journals,
as well as portfolios of learning evidence (Van Tonder, 2010:19). The principles of credible
assessment will at some point differ between conventional and alternative assessment, and it
is therefore necessary to note that the data for this study is considered to be a form of
conventional assessment.
2.4.2 Kinds of knowledge
Biggs and Tang (2007:72) emphasise that knowledge is the object of understanding (which
can be structured into levels) and that there are two main kinds of knowledge in the context of
assessment. These two main kinds of knowledge are stated as declarative knowledge or
functioning knowledge (Biggs & Tang, 2007:72). In linking to the above discussion it is
therefore also important to consider whether an assessment aims to cover declarative
knowledge or functioning knowledge.
Biggs and Tang (2007:81) describe declarative knowledge as knowing about phenomena,
disciplines and theories (e.g. “what Freud said”). It is content knowledge that accrues from
others’ research, not from personal experience, and is typically found in libraries, textbooks,
and what teachers declare in lectures (Biggs & Tang, 2007:72). On the other hand, Biggs and
Tang (2007:72) explain that functioning knowledge is based on the idea of performance
underpinned by fundamental understanding (e.g. how to physically fly an aeroplane). This
knowledge is within the experience of the learner, who can now put declarative knowledge to
work by solving problems such as performing surgery. Functioning knowledge requires a solid
35
foundation of declarative knowledge (Biggs & Tang, 2007:72). Whether an assessment will
cover declarative knowledge or functioning knowledge will depend on its context. For financial
accounting as a professional discipline, the assessment process is likely to lean more towards
functioning knowledge. Fortunately, due to the inherent nature of financial accounting (e.g.
supporting documentation, standards-based reporting and working with numbers) a traditional
assessment such as an exam can simulate real-life scenarios in order to assess functioning
knowledge to a large extent. This may not be the case for other professionals such as pilots
or surgeons. The SOLO taxonomy can assist in the assessment process of both declarative
as well as functioning knowledge (see paragraph 2.5.1.3 below).
2.4.3 Forms of assessment
According to Biggs and Tang (2007:170) another set of concepts that influences current
practice (and therefore needs discussion) is the distinction between assessment in the form
of norm-referenced assessment and assessment in the form of criterion-referenced
assessment.
Norm-referenced assessment (NRA) is described as assessment with the aim of
differentiating between students based on their results (Biggs & Tang, 2007:179), hence the
focus on a “normal” distribution of results. Criterion-referenced assessment (CRA) in
contrast entails assessment with the aim of judging how well a student’s performance matches
a criterion that has been set in advance (Biggs & Tang, 2007:179). NRA therefore makes
judgments about people whereas CRA makes judgments about performance (Biggs & Tang,
2007:179). For CRA whether a group of students all score full marks or very low marks are
inconsequential, since the issue is whether the outcomes were evidently met or not. Biggs and
Tang (2007:170) admit that there may be instances where a normative approach is applicable,
such as research involving measurement along a graduated scale so as to compare
individuals to each other or against population norms. The use of NRA to assess educational
outcomes is however not appropriate (Biggs & Tang, 2007:170). Amongst others, this is
because NRA is highly likely to be unfair assessment due to it being from of bias (e.g. leaners
may fail because of relative marks while they did achieve the ILOs). Meanwhile, SAQA
(2014:6) essentially describes (and demands) fairness in assessment as entailing the
absence of any bias (e.g. also gender, etc.) and that assessment tasks are aligned to teaching
activities and learning outcomes.
NRA aims to assess from the viewpoint of obtaining a normally distributed set of grades (Biggs
& Tang, 2007:180) instead of from the viewpoint of the agreement with the learners (i.e. the
curriculum and ILOs). NRA is likely to lead to accusations of bias. An example of how NRA
36
comes about is the pre-occupation by some managers with exam grade averages, which
sometimes lead to the request for an “easier” of “more difficult” exam in order to “normalise”
the distribution of marks, whereas the question should rather have been “Have the students
demonstrated that they have achieved the learning outcomes?” Biggs and Tang (2007:170-
179) link CRA to the desired context of constructive alignment (paragraph 2.2), deep learning
(paragraph 2.3.1), and the SOLO taxonomy (paragraph 2.5.1).
2.4.4 Taxonomies
The fundamental principles of constructive alignment, deep learning and CRA are some of the
main reasons that teachers and assessors make use of taxonomies of learning to facilitate
these processes. Taxonomies of learning, such as the Bloom’s Taxonomy (see paragraph
1.1.1 above) provides a framework to measure students’ responses (i.e. their thinking and
learning) but is also used for the design of ILOs (Biggs & Tang, 2007:26) and to evaluate
assessment (paragraphs 1.1.1 and 1.1.2).
2.4.5 Principles of credible assessment
The discussion above inherently illuminates some main principles for credible assessment,
but in order to provide a further well-grounded context for the use of the SOLO taxonomy,
Brown’s (2001:6) as well as the IAESB’s (2019:82-84) guidance will now be summarised.
Brown’s (2001:6) principles are succinctly given and are stated as:
assessment shapes learning, so if you want to change learning then change the
assessment method;
match the assessment tasks to the learning outcomes;
match the criteria to the task and learning outcomes;
keep the criteria simple;
be fair, reliable and valid in your marking;
provide meaningful, timely feedback.
In essence, Brown’s principles show that he recognises that the “backwash” effect of
assessment (first bullet) should be positively used, as well as that he supports constructive
alignment (last five bullets).
The IAESB (2019:77) also provides conceptual guidance for assessment of professional
competence related to accounting education in the standard IES6. IES6 (IAESB, 2019:79)
initially lists three overall requirements for assessment by IFAC member bodies (see
paragraph 2.2) after which explanatory guidance is given. These three overall requirements
are:
37
i) the need for formal assessment;
ii) assessment activities that “have high levels of reliability, validity, equity,
transparency, and sufficiency within professional accounting education programs
[sic]”; and
iii) that “IFAC member bodies shall base the assessment of the professional
competence of aspiring professional accountants on verifiable evidence”.
The application of the SOLO taxonomy, which is the subject of this study, fits into the broader
scheme of universities having to comply with these three requirements if their financial
accounting programmes are to remain accredited with a professional body who is a IFAC
member. For purposes of relevance and clarity the second overall requirement will now be
discussed.
IES6 (IAESB, 2019:82) requires that assessment activities should always aim to have a high
degree of reliability. Reliability of assessment is acknowledged as being contextual and not
absolute, but is described as assessment activities where assessors (or examiners) will
independently for a given set of circumstances come to the same judgement about the
assessment generated evidence. In essence, this requirement is also one of fairness and lack
of bias. Reliability can be increased by, for example, using clear and unambiguous language
in an examination and by using internal and external moderators to review an assessment.
IES6’s (IAESB, 2019:82) requirement that assessment activities should aim to have a high
degree of validity means that an assessment activity should measure what it was intended to
measure. This should be perceived to be the case (face validity) and indeed be the case
(predictive and content validity). Validity of assessment is also acknowledged as being
contextual and not absolute. What IES6 is essentially advocating here is the use of
constructive alignment, since constructive alignment entails that ILOs, teaching and learning
activities and assessment tasks are all aligned, which means that an assessment activity will
measure what it intended to measure. An example that IES6 (IAESB, 2019:83) gives of
increasing validity is when an examination includes a comprehensive and relevant case study
rather than a simple case study based on incomplete information.
An IES6 (IAESB, 2019:83) requirement that assessment activities should always aim to have
a high degree of equity means that the activities should be fair and without bias. Equity is also
acknowledged as being contextual and not absolute. An example of raising equity is the
removal of assumptions relating to cultural knowledge that are not commonly shared by all
aspiring professional accountants from an examination.
38
An IES6 (IAESB, 2019:83) requirement that assessment activities should always aim to have
a high degree of transparency means that details of an assessment activity, such as the
competence areas to be assessed and timing of the activity, are disclosed publicly.
Transparency is also acknowledged as being contextual and not absolute. The given example
of raising transparency is the use of clear and accessible communications to stakeholders.
For this researcher the disclosure of ILOs and assessment criteria in a system of constructive
alignment is also a great way to enhance transparency.
The final part of this set of requirements is that assessment activities should always aim to
have a high degree of sufficiency (IAESB, 2019:84). An assessment activity has a high level
of sufficiency if it: i) has a balance of detail and scope, around knowledge and application; and
ii) provides for a range of situations and contexts. The use of the SOLO taxonomy in evaluating
an assessment should greatly assist with this requirement, since it will guide an examiner in
considering a range of cognitive demands which would point out a lack or surplus of detail or
different contexts. Sufficiency is also acknowledged by IES6 as being contextual and not
absolute. The example that is given by IES6 (IAESB, 2019:84) in terms of sufficiency is the
inclusion of assessment activities that assess technical competence, professional skills, and
ethics, and is set at an appropriate level of detail.
In conclusion, one must admit that there are many more concepts around assessment as well
as books full of research and advice around this cardinal topic. For this study, the above
discussion is sufficient in expanding on the specific context of assessment of financial
accounting in which the SOLO taxonomy will be applied.
2.5 CONCEPTUALISING THE APPLICATION OF THE SOLO TAXONOMY IN
EVALUATING THE LEVEL OF SUMMATIVE ASSESSMENTS
2.5.1 Exploring the concept of SOLO
2.5.1.1 Origination of the SOLO taxonomy
Pamela Hook (2015:1), who is a leading authority in classroom teaching in New Zealand,
states that the SOLO taxonomy began as an evidence-based model by university academics
Biggs and Collis in Australia in the late 1970s (Hook, 2015:6). She notes that it was developed
from research on samples of students’ thinking. Biggs and Tang (2007:81) confirm that SOLO
was based on research of student learning itself which they see as an advantage over the
Bloom and the revised Bloom’s taxonomies. The SOLO taxonomy also features prominently
on Bigg’s online blog (2019).
39
Both Biggs and Tang (2007:21) and Lucas and Mladenovic (2009:262) acknowledge that the
SOLO taxonomy is partly rooted in the theory and work of pioneering psychologist Jean
Piaget. Piaget’s famous work proposed four stages of cognitive development which reflect the
increasing sophistication of children's thought, namely the sensorimotor, preoperational,
concrete operational and the formal operational stages of cognitive development (McLeod
2018:7). Figure 2-2 (page 39) provides an overview of these stages as an illustration of the
SOLO taxonomy’s “roots”.
Figure 2-2: The “roots” of the SOLO taxonomy: Piaget’s four stages of cognitive development of
children's thought
Source: McLeod (2018:8-9)
2.5.1.2 The meaning of the SOLO taxonomy
The SOLO taxonomy, which stands for structure of the observed learning outcome, provides
a systematic way of describing how a learner’s performance grows in complexity when
mastering many academic tasks (Biggs & Tang, 2007:76). Biggs and Tang (2007:76) states
that the SOLO taxonomy was, amongst others, built on the premise that “understanding” (i.e.
the cognitive process) is something that develops gradually, becoming more structured and
articulated as it develops. This researcher’s experience agrees with this statement.
The SOLO taxonomy is shown as a useful tool that provides structure to observe qualitative
data such as learning outcomes (or assessment tasks or the actually intangible concept of
“understanding”). Lucas and Mladenovic (2009:263) define SOLO as follows:
1. Sensorimotor Stage (Birth-2 yrs) 2. Preoperational Stage (2-7 years)
Object permanence - knowing that an object
still exists, even if it is hidden.
It requires the ability to form a mental
representation (i.e., a schema) of the object.
Think about things symbolically. This is the ability to
make one thing - a word or an object - stand for
something other than itself.
Thinking is still egocentric, and the infant has
difficulty taking the viewpoint of others.
3. Concrete Operational Stage (7-11 years) 4. Formal Operational Stage (11 years and over)
The beginning of logical or operational thought.
The child can work things out internally in their
head (rather than physically try things out in the
real world).
Children can conserve number (age 6), mass
(age 7), and weight (age 9). Conservation is the
understanding that something stays the same
in quantity even though its appearance
changes.
People develop the ability to think about abstract
concepts, and logically test hypotheses.
40
“… The SOLO taxonomy thus provides a normative structure for the identification of
variation in students’ understanding of disciplinary concepts …”
Biggs and Tang (2007:76) developed SOLO on the premise that, as students learn, the
outcome of their learning should display increasing structural complexity, hence SOLO is
structured in levels. This premise which, is also known as “scaffolding”, is widely adopted in
formal education with the pertinent example being the NQF with its level descriptors (SAQA,
2012).
Biggs and Tang (2007:76) mainly advocate SOLO as a means to define and evaluate learning
outcomes in the context of constructive alignment. It is thus to be expected that most other
studies demonstrate SOLO as a very useful tool for measuring the level of students’ responses
in light of the relevant learning outcomes. An example of this usage is found in Newton and
Martin’s (2013:78) study that relates to biochemistry.
Ramsden (in Biggs & Tang 2007:xvi), however, emphasised the practical utility of SOLO as
being an assessment apparatus in itself. As shown in paragraph 2.5.2 below, a review of
studies utilising SOLO in one form or the other, demonstrates the utility of SOLO. The logical
question that followed this point (of SOLO’s utility) is whether SOLO can be used to evaluate
an assessment or collection of assessment tasks themselves (see paragraph 1.1.2).
Therefore, the objectives of this study were set (paragraph 1.3) so as to demonstrate that
SOLO is not only a tool to evaluate students’ responses as an indication of their
understanding, but also to evaluate the level of response an assessment requires from a
student, and hence set a framework for evaluating an assessment’s level.
As mentioned in paragraph 1.1.1, the SOLO taxonomy is structured into five levels, which are
discussed below.
2.5.1.3 The SOLO taxonomy’s levels
Summarising and synthesising from Biggs and Tang (2007:77-79) and Hook (2015:5), SOLO’s
levels are described below and a visual summary is given in Figure 2-3 (page 44). Biggs and
Tang (2007:77-79) also suggest some (not all) verbs as words that typically indicate (but still
needs careful consideration) a level of the SOLO taxonomy and which can aid in determining
the level of a learning outcome or assessment task. One should note that these verbs on their
own may still be too generic for evaluation of an introductory financial accounting assessment
(this pertinent problem is discussed in paragraph 1.1.2 above and is taken up as part of the
research objectives per paragraph 1.3.2).
41
I. Pre-structural: A response that is “missing the point”. The response is irrelevant, which
shows a lack of understanding. It should be noted that the response can be
sophisticated utilising “elaborate tautology”, but since it is irrelevant, it misses the point.
Hook (2015:5) summarises this level as one where the “learner” has “no idea”.
Verbs (Biggs & Tang, 2007:80): There are no verbs that are specifically linked to this
level, since this level points to irrelevant responses and a total lack of understanding.
Further comment: Interestingly, Biggs and Tang (2007:77) and Biggs (2019) note that
politicians that use an elaborate tautology to avoid answering questions are functioning
on this lowest of SOLO’s levels, since academically they show little evidence of
relevant learning. For this study’s purpose it would be surprising to find an assessment
task requiring and awarding such a response.
II. Unistructural: For this level, the quantity of knowledge required increases but not the
qualitative level of understanding. Responses typically deal with terminology.
Responses deal with one concept, superficial and in isolation. Hook (2015:5)
summarises this level as one where the “learner” has “one idea”. This level can thus
also be described as being one-dimensional.
Verbs (Biggs & Tang, 2007:80): Memorize, identify, recognize, count, define, draw,
find, label, match, name, quote, recall, recite, order, tell, write, imitate.
Further comment: One should not by default dismiss this level as “bad” or “useless”. It
may well be appropriate depending on the context of learning, e.g. one needs to
memorise a letter in the alphabet before one can differentiate it from other letters.
Lucas and Mladenovic (2009:262) support this notion that the cognitive response to
be required is driven by the context of learning.
III. Multi-structural: Student responses (or the assessment tasks’ required responses)
deals with several concepts (i.e. more than one) but they are unrelated. The increase
in level is still quantitative in nature. Hook (2015:5) summarises this level as one where
the “learner” has “several ideas”. This level can also be described as being two-
dimensional.
Verbs (Biggs & Tang, 2007:80): Classify, describe, list, report, discuss, illustrate,
select, narrate, compute, sequence, outline, separate.
42
Further comment: This level is an important building block to the higher levels.
Learning outcomes or assessment tasks that is structured around this level can be
seen as foundational in nature.
IV. Relational: Biggs and Tang (2007:76) state that the SOLO levels initially increase
quantitatively (amount of detail) and then qualitatively (integration of detail into a
structural pattern). This level marks the defining point where the qualitative phase
becomes paramount. In other words, it is the level where the aim is the deepening of
understanding as opposed to the mere increase in knowledge (Biggs & Tang,
2007:79). In this level, student responses (or the assessment tasks’ required
responses) deal with several concepts and the response is able to show how these
concepts relate to each other as well as to a topic as a whole. Hook (2015:5)
summarises this level as one where the “learner” has “related ideas”, with connections
being made and parts synthesised with overall meaning. This level therefore also
entails the integration of different subjects’ matter, e.g. financial accounting and
taxation.
Verbs (Biggs & Tang, 2007:80): Apply, integrate, analyse, explain, predict, conclude,
summarize (précis), review, argue, transfer, make a plan, characterize, compare,
contrast, differentiate, organize, debate, make a case, construct, review and rewrite,
examine, translate, paraphrase, solve a problem.
Further comment: Since this level deals with at least two concepts and their relation, it
can be described as being three-dimensional in nature. An important point in this level
is that for a professional discipline such as financial accounting, the ability to relate
concepts means that knowledge becomes functioning as opposed to just being
declared. Biggs and Tang (2007:81) describe functional knowledge as the exercise of
active control over problems and decisions in the appropriate content domains (also
see paragraph 2.4.2). Biggs and Tang (2007:73), who note the work of Leinhardt et al.
(1995) as well as Entwistle and Entwistle (1997), tellingly state that would-be
professionals are trained in universities to label, differentiate, elaborate and justify,
when what they need out in the field is to execute, apply and prioritize. This statement
would suggest another reason why the SOLO level of assessment needs to be
carefully considered.
V. Extended abstract: According to Biggs and Tang (2007:78), in this level student
responses (or the assessment tasks’ required responses) go beyond what has been
given. The coherent whole is conceptualized at a higher level of abstraction and is
43
applied to new and broader domains. An extended abstract response includes what
would be a breakthrough response, giving a perspective that changes what we think.
In essence, this level of cognitive demand is the ability to use the known tools to solve
the problems not encountered before. Hook (2015:5) summarises this level as one
where the “learner” has “extended ideas” by going beyond the subject and making links
to other concepts. This level can therefore also be described as being four-
dimensional, since other concepts that are new or not obviously related are now also
linked to the familiar concepts.
Verbs (Biggs & Tang, 2007:80): Theorize, hypothesize, generalize, reflect, generate,
create, compose, invent, originate, prove from first principles, make an original case,
solve from first principles.
Further comment: This level is highly valuable in that new knowledge is typically
created here, as well as that it demonstrates an adaptability towards problem solving.
It shows the ability to deal with the unknown or unforeseen, i.e. a high form of problem
solving. However, one should guard in over-emphasising this level or considering it to
necessarily be of more worth than the other SOLO levels. As explained in the following
paragraphs, this level still forms part of a broader context of the learning experience.
Biggs and Tang (2007:78) note that today’s extended abstract is tomorrow’s relational,
as new knowledge becomes conventional wisdom. It therefore follows that each
person’s application of extended abstract will be different according to their experience
and training. The issue in terms of a professional environment such as around financial
accounting, is that there is not necessarily the need or resources such as time and
funding available to a professional person to apply extended abstract reasoning.
Although it is quite logical that a professional should be able to “think on her feet”, one
would not want the professional person to “reinvent the wheel” either. A good
theoretical foundation as well as functional knowledge are thus essential to a financial
accounting student, so as to ensure that “the wheel does not need to be reinvented”.
This functioning knowledge would however include the ability to operate on an
extended abstract level.
Biggs and Tang (2007:79) also make it clear that each SOLO level incorporates the
previous levels and is thus a foundation (or “scaffold”) on which further learning is built
(or demonstrated in the case of an assessment). This resonates with a well-known
teaching strategy of “building on the known to help explain the unknown”.
44
Figure 2-3: A visual summary of SOLO
Source: Adapted from Biggs and Tang (2007:79) and Hook (2015:5)
2.5.2 Review of other scholarly research on SOLO
In paragraph 1.4.1 above, one of the functions of the literature review was explained as to
build on existing scholarship and knowledge. This includes the discovering of recent theorising
around the subject, consideration of widely accepted empirical findings in the field of the study,
and familiarisation with the widely accepted definitions of key concepts in the field of the study
(Mouton, 2013:87). Although these functions are addressed in the different paragraphs of this
chapter, as a whole this section will specifically focus on the latest findings of other scholars
as reported in academic journals. As stated in paragraph 1.2 above, there is a gap in the
literature that deals with all of i) the SOLO taxonomy, ii) assessment and iii) financial
accounting. Therefore, the literature that was found was considered if at least two of these
three core components were present in a study. For purposes of relevance, impact and
conciseness, the literature found to be closest to this study in terms of the topic will be
discussed. This includes the two studies mentioned in Table 1-2 (page 17) above.
2.5.2.1 Lucas and Mladenovic’s study
As mentioned in paragraph 1.1.2 above, the study by Lucas and Mladenovic (2009) also
investigates issues around the topics of SOLO and introductory financial accounting, and is
45
the literature found to be the closest to this study in terms of the topic, hence it will be
discussed first. Some of their findings were also mentioned in paragraph 2.4.2 above.
Lucas and Mladenovic (2009:263) point out that they agree with prior research that shows that
students under stress tend to revert to a lower level of cognitive response. In the experience
of this research (in line with Biggs and Tang, 2017:199) the issue of time allowed to complete
an assessment task is inherently such a stressor. In paragraph 4.3.1 below, the time allocation
for each examination’s assessment tasks is investigated.
Lucas and Mladenovic (2009:277) also note that an inherent limitation of their study is the
element of subjectivity during allocation of student responses to SOLO levels. They do
however explain that they managed this limitation in terms of their study’s objectives
(2009:278). The management of this subjectivity in terms of evaluating assessment tasks is
one of the objectives of this study.
Lucas and Mladenovic’s (2009:277) overall conclusion is that the understandings evidenced
by the majority of introductory financial accounting students may not meet the expectations
within higher education. In other words, it can be said that the students do not operate on the
expected SOLO level. Lucas and Mladenovic (2009: 276-277) then suggest that there should
be interventions to support changes in teaching and learning approaches, including
assessment. The nature of such interventions is discussed in broad terms, and it is mentioned
that these should involve the use of SOLO as a diagnostic tool for both teachers and leaners,
to find where the problems in understanding may be (an example of such a problem is the
occurrence of misunderstandings or misconceptions). The detail of such interventions, such
as the issue around curriculum development, was deferred as part of their future research
agenda.
In terms of Lucas and Mladenovic’s (2009) study and assessment as a concept, one should
again highlight that it is crucial to ensure constructive alignment of assessment to the learning
outcomes and other teaching and learning activities. This is so that any changes in the
teaching and learning approach flow through to assessment in order for the total system to
achieve its objectives.
2.5.2.2 Newton and Martin’s study
The study of Newton and Martin (2013) was found to be second most relevant in terms of this
study’s topic. In terms of Newton and Martin’s (2013) study, student responses in an exam of
an undergraduate biochemistry course was evaluated in terms of two conceptual frameworks,
i.e. a phenomenography derived from their analysis of student responses as well as the SOLO
46
taxonomy. The exam’s questions themselves were ranked according to their level on the
Bloom’s taxonomy (Newton & Martin, 2013:78). The objective of the investigation was to
determine the relationship between students’ exam responses (relating to biochemistry)
across two hierarchical categories (being SOLO and their own phenomenography) and the
students’ learning approach as measured by scores on the R-SPQ-2F (Newton & Martin,
2013:79-80). Newton and Martin (2013:78) explain that the R-SPQ-2F is the Revised Student
Process Questionnaire 2-Factor, which is commonly used to measure students’ learning
approach in terms of deep or surface learning.
The overall results of Newton and Martin’s (2013:88) study confirm the relationship between
the deep learning approach and improved performance at higher levels of cognitive complexity
(i.e. the higher levels on the Bloom’s or SOLO taxonomy or their own phenomenography), and
that alternative assessment strategies (such as open-ended questions or higher grades for
higher level approaches) may be valuable tools with which to evaluate learning at increasing
levels of cognitive complexity.
For the purposes of this study, the following can also be seen from Newton and Martin’s (2013)
study:
more evidence that taxonomies such as Bloom’s and SOLO can be used in terms of
students’ responses as well as assessment;
examiners, in using SOLO with assessment, should carefully consider the design of
the assessment tasks so as to accommodate a range of levels of responses to be
appropriate (Newton & Martin, 2013:88); and
that the design of assessments can be used as a tool to encourage students to operate
at higher cognitive levels (Newton & Martin, 2013:88).
The last two bullets above provide additional confirmation of why the evaluation of an
assessment through the SOLO taxonomy may be particularly useful.
2.5.2.3 Other studies
In conclusion of the section it is noted that further academic articles indicated a point of
saturation for this review in that they covered the various iterations of using the SOLO
taxonomy to evaluate students’ responses (e.g. during entry or final exams), to evaluate
students’ learning (i.e. as deep or surface) or to evaluate learning outcomes. Slack et al.’s
(2013) study titled “Assessment and learning outcomes: The evaluation of deep learning in an
on-line course”, can be seen as an example.
47
2.6 SUMMARY
In this chapter, a systematic literature review was undertaken, which included i) addressing
the first two research objectives of contextualising summative assessment (paragraphs 2.2 -
2.4) as well as conceptualising the application of the SOLO taxonomy (paragraph 2.5), and ii)
summarising and building on the work of other scholars (paragraph 2.5.2).
48
CHAPTER 3 RESEARCH DESIGN AND METHODOLOGY
3.1 INTRODUCTION
In Chapter 2 above, a systematic literature review was undertaken which included
i) addressing the first two research objectives of contextualising summative assessment and
conceptualising the application of the SOLO taxonomy, and ii) summarising and building on
the work of other scholars.
Keeping the overall objective of this study in mind, i.e. to provide a framework for the
application of the SOLO taxonomy in evaluating the standard of summative assessment of a
Financial Accounting 1 module (as stated in paragraph 1.3.1), the research design and
methodology (including paradigmatic assumptions) of this study will be motivated and
defended here in the sections that follow. A thorough motivation of the research design and
methodology is especially necessitated by the process of gathering and analysing the
empirical evidence that is needed to achieve research objectives nr. 3 to nr. 5 (see paragraph
1.3.2), as well as to obtain credibility for the study. The research design and methodology
described in this chapter will then later be effected in Chapter 4. The discussion that follows
is started on an initial broad level by explaining what the concept of “research” entails.
A general definition of “Research” is that it is an activity that involves a systematic way of
expanding knowledge (Walliman, 2011:7). Performing and documenting a systematic process
is closely linked to the rhetoric, validity and credibility of the research. There is arguably little
point in research if it is not credible and convincing. The systematic process of research has
been developed, refined and illustrated by many researchers over a long period of time and
to a point where the minimum elements of the process are well accepted. The aforementioned
statement is easily verified by just noting the index of many sources such as Mouton (2013),
Cohen et al. (2011), Smith (2011), Walliman (2011), Saunders et al. (2016), to name but a
few.
A well-illustrated way of the systematic process of research is the framework called the
“research onion” which was developed by Saunders et al. (2016:124). As can be seen in
Figure 3-1, this framework starts with the philosophies of research at the outermost layer and
then moves on to the research approaches, strategies, choices and time horizons, and finally
ending with techniques and procedures at the centre. The following sections of this chapter
will draw on the “research onion” in order to document the systematic process followed for this
study. Some of the aspects that were discussed in Chapter 1 as part of the planning phase of
the study will be included here on a more comprehensive and applied basis.
49
Figure 3-1: The research “onion”
Source: Saunders et al. (2016:124)
3.2 PHILOSOPHIES OF RESEARCH
All research is based on assumptions of how the world (or “reality”) is perceived and how one
can best come to understand it (Trochim & Donnelly, 2008:14). The aforementioned is also
known as the “philosophy of research”, which can also be stated as “a system of beliefs and
assumptions about the development of knowledge” (Saunders et al., 2016:124). Over the
years scholars have structured their different research philosophies into research paradigms
that groups a constellation of beliefs, values and techniques (Kuhn as cited by Burke, 2007:3;
Clark, 1998:1243; Cohen et al., 2011:5). The research assumptions underpinning each
paradigm are grouped as being part of ontology, epistemology and axiology (Saunders et al.,
2016:127) which are discussed below. Closely related to the epistemological assumptions,
but focusing more on the practical side of development of knowledge, are the methodological
assumptions (Trochim & Donnelly, 2008:18).
A research paradigm can thus be seen as a (widely accepted) lens through which the
researcher views the world (Henning, 2005:2; Burke, 2007:4). It is a way for a researcher to
frame his work as well as defend his research (Burke, 2007:4). The paradigm assists in
framing assumptions and context for the researcher as well as to other scholars (Burke,
2007:1). Bertram and Christiansen (2017:22) agree with the above stance as they explain that
although the term “paradigm” has been used in different meanings and contexts by different
50
authors, it is widely accepted that it refers to the worldview that defines what is acceptable to
research and how it should be done. This includes the issues of what can be observed and
investigated and how data are collected and interpreted. Mouton (2013:141) has developed a
useful tool that shows that the research paradigm (stated as “World 3” or Meta-sciences) links
to the methodological approach (stated as “World 2” or Science) which in turn is influenced by
the real everyday world experience (stated as “World 1” or the Pragmatic interest).
A discussion of major research paradigms and assumptions as well as their application to this
study follows, but before the discussion continues it is important to note that accounting
researchers (including accounting education) adapt their theory from the natural and social
sciences and thus may heavily rely on adapting the instruments of other disciplines (Smith,
2011:1).
From the literature (Saunders et al., 2016:124; Van Rooyen, 2016:19; Cohen et al., 2011:6) it
can be seen that the work of Burrell and Morgan (1979) regarding philosophises for research
in social sciences still prevails, but was obviously further developed and built on by other
scholars. An example of this would be the positivist paradigm which corresponds to Burrell
and Morgan’s functionalist paradigm (Burke, 2007:8). Thus, when it comes to paradigms,
scholars are broadly in agreement as to the major categories but often use slightly different
terminologies or expanded categories (see Table 3-1, page 51).
The two major paradigms or schools of thought being evident is positivism and interpretivism
(Grey, 2014:21). Grey (2014:21) also states that “… of the different theoretical perspectives
available, positivism and various strands of interpretivism are, or have been (arguably) among
the most influential”. Some authors, however, start their discussion of philosophy with a
discussion of positivism and post-positivism (Trochim & Donnelly, 2008:14). They see post-
positivism as a paradigm on its own. Trochim and Donnelly (2008:19) for example describe
post-positivism as the rejection of positivism in favour of inferences about phenomena based
upon theoretical reasoning combined with evidence based on experience. Cohen et al.
(2011:27) give credibility to this view by stating that depending on the particular view of post-
positivism, knowledge is regarded as subjective rather than objective. Although this
researcher agrees with these broad definitions, the view taken is one of “post-positivism”,
meaning “after” or “further” than positivism. Burrell and Morgan (1979:3) refer to this concept
as “anti-positivism”.
From Table 3-1 and in line with Grey (2014:21), positivism and interpretivism can be seen as
two of the most pertinent paradigms. This is no surprise, since these two philosophes form the
cornerstones of Burrell and Morgan’s (1979:22) paradigms model. Positivism and
51
interpretivism stand on the opposite side of the objectivism/subjectivism and
deductive/inductive reasoning divide (see discussion below). These two paradigms will
therefore constitute a main discussion and will be accepted or rejected, while pragmatism will
also be explored due to its prominence for problem-based research (Saunders et al.,
2016:137) and foreseeable applicability to this study.
Table 3-1: The typical paradigms discussed by scholars
Author Paradigms discussed
Trochim and Donnelly, 2008:14 Positivism Post-Positivism
Saunders et al., 2016:135-134 Positivism Anti-Positivism:
Critical realism
Interpretivism
Postmodernism
Pragmatism
Cohen et al., 2011:47 Positivism Anti-Positivism:
Interpretivism
Critical theorist
Smith, 2011:5 Positivism Anti-Positivism:
Interpretivism
Critical theorist
Henning, 2005:17-23 Positivism Anti-Positivism:
Interpretivism
Critical theorist
Mouton, 2013:140-141 Positivism Anti-Positivism:
Interpretivism
Critical theorist
Realism
Phenomenology
Postmodernism Source: Researcher
As mentioned above, the distinction between objectivism and subjectivism is a major anchor
of the different research philosophies. Objectivism or subjectivism is a function of the
scientist’s ontological stance (Burrell & Morgan,1979:3). Objectivism typically incorporates the
assumptions of the natural sciences, arguing that the social reality that we research is external
to us and others (Saunders et al., 2016:128). Objectivism is in essence the cornerstone of the
positivist paradigm where the researcher stands independent of a universal reality (Cohen et
al., 2011:46). In contrast to this, subjectivism is typically the ontological stance of the
interpretive paradigm and asserts that social reality is made from the perceptions and
consequent actions of people (Saunders et al., 2016:130). In essence, subjectivism accepts
the researcher as an insider of the process of creating meaning while it is accepted that there
52
may be multiple realities or truths (Cohen et al., 2011:46). Throughout this study a subjective
stance will be seen, explained and drawn upon.
Saunders et al. (2016:134) describe positivism as the philosophical stance of there being an
observable social reality that produces law-like generalisations. Saunders et al. (2016:124)
attribute this stance as being typical of the natural scientist. Trochim and Donnelly (2008:18)
agree with Saunders et al. (2016:135) in that positivism holds that the purpose of science is
to stay at what can be objectively observed and measured. Cohen et al. (2011:46) list
attributes such as normative in approach, linear causality, structuralism, outsider perspective,
quantitative and objective in nature to the positivist paradigm. In essence then, the positivist
stance is one where there is an observable, independent and objective reality or truth that can
be deductively measured and generalised from.
The main reason that the positivist stance is rejected for this study can be found in Saunders
et al. (2016:135), who state that positivism focusses on data and facts that are uninfluenced
by human interpretation or bias. It is clear from this study’s research problem and questions
(as stated in paragraph 1.2 and 1.3) that the data will be qualitative in nature and will be
interpreted to obtain meaning. That is, Financial Accounting 1 exams will be analysed to,
amongst others, link assessment tasks to the SOLO levels. This process is interpretive in
nature.
The central endeavour of the interpretive paradigm is to understand the subjective world of
human experience (Cohen et al., 2011:17). Saunders et al. (2016:140) add to this by stating
that interpretivism entails the studying of meaning as created by humans.
In light of the reasons discussed above, as well as the further justification given in Table 3-2,
this study is framed predominantly in the interpretive paradigm. It is, however, argued below
that a smaller secondary but material part of the study leads to the researcher eventually
drawing on the research philosophy of pragmatism.
Table 3-2: Further justification of the applicability of the interpretive paradigm
Tenets of the interpretive paradigm Relevance to this study
The interpretive paradigm confronts the
difficulties presented by the nature of the
research domain such as the intangibility of many
of the factors and relationships (Clarke in Burke
There may be a myriad factors which interact and
lead to an assessment being perceived to have a
certain level of cognitive demand. Many of these
factors, including the concept of cognitive
demand in itself, are incorporeal in nature.
Furthermore, this study does not aim to provide
53
2007:11 and agrees with Cohen et al., 2011:46
and Saunders et al., 2016:136).
an exact recipe of how many factors provide
which exact degree of cognitive demands, but
aims to, through interpretation of the assessment
tasks and the SOLO taxonomy, formulate a
framework that can guide the application of the
SOLO taxonomy in evaluating the standard of
summative assessment of a Financial
Accounting 1 module.
The inherent involvement of the researcher
within the research domain; the dependence of
outcomes on the researcher’s perspective, such
as the definition of the research question, the
design of the research question and the
measurement of variables (Clarke in Burke
2007:11 and agrees with Cohen et al., 2011:46
and Saunders et al., 2016:136).
In Chapter 1 it can be seen that the researcher
has set the research questions and is centrally
involved in the interpretation of the data.
Multiple meanings, interpretations, realities. Flux
of processes, experiences, practices. (Saunders
et al., 2016:136 and agree with Cohen et al.,
2011:17,27,46).
In Chapter 1 it is described that this study is set
in a constantly changing world where, in the light
of the research questions, there may be multiple
answers or realities. In this study this researcher
creates further meaning by interpreting what the
assessment tasks mean in light of the SOLO
taxonomy.
The interpretivist is to enter the social world of the
research participants and understand that world
from their point of view (Saunders et al.,
2016:141 and agree with Cohen et al., 2011:46).
This study utilises a form of archival or
documentary data (financial accounting exams)
with the researcher being the main research
participant. However, in order to analyse the
exams and in order to create meaning when
analysing, the researcher will draw on his own
experience as an accountant, lecturer and former
Financial Accounting 1 student to place himself
as a current student writing the exam, so as to
obtain a richer understanding of how the
assessment tasks and the SOLO taxonomy
relate.
54
Typically inductive. Small samples, in-depth
investigations, qualitative methods of analysis
(Saunders et al., 2016:136 and agree with Cohen
et al., 2011:46).
It can be seen in Chapter 1 as well as in the other
sections of this chapter that these attributes are
applicable to this study.
Source: Researcher
According to Saunders et al. (2016:151), pragmatist ontology, epistemology and axiology are
focused on improving practice. Pragmatism reconciles the use of a range of methods (e.g.
quantitative, qualitative and mixed methods), theories and strategies with the aim of
contributing practical solutions that inform future practice (Saunders et al., 2016:143).
Pragmatism as a philosophy especially finds application where the research methods to be
chosen will be mainly driven by the research problem and questions (Saunders et al.,
2016:137). Morgan’s (2014:7) stance supplements this view in stating that pragmatism allows
for research that is based on the beliefs and actions of researchers rather than a fixed
commitment to an abstract set of philosophical beliefs.
By the above explanations it can be seen that although a pragmatic philosophy includes
positivist approaches, it is essentially a “post-positivist” paradigm (as explained above) since
it accommodates a range of philosophies, with the overall aim being one of solving the
research problem, which typically has practical interest.
Saunders et al. (2016:143) warn scholars not to use pragmatism as an escape route from
understanding other research philosophies. This makes sense, especially where a pragmatic
approach draws on, and includes, assumptions from other philosophies as part of the bigger
project of addressing the research problem, questions and objectives.
During the reasoning above and as indicated in Table 3-2 (page 52), it was motivated that this
study is predominantly framed in an interpretive paradigm. The study is, however, also clearly
concerned with addressing a defined research problem and objectives (see Chapter 1 above).
Therefore, this researcher does not wish to limit the possibilities of the research methods and
data analysis to the assumptions accepted under the interpretative paradigm, since, i) as
explained in paragraphs 1.4.2 and 3.4.1, the study does have some quantitative data and ii)
the data analysis process of this study (as described in paragraph 3.4.2) will entail some
quantitative techniques. The quantitative data and analysis will be secondary, basic and
relatively small in scope but is important enough so as that overall this study is set in the
pragmatic paradigm as materially underpinned by the interpretative paradigm.
55
During the initial phase of this study the ontological and epistemological assumptions as
pertaining to the primarily interpretivist approach are explained and described (paragraph
1.4.3). Since this study is ultimately set under a pragmatic paradigm (which makes provision
for inclusion of the interpretivist approach), the ontological, epistemological and axiological
assumptions are further summarised in Table 3-3.
Table 3-3: Assumptions under pragmatism
Group of assumptions Short definition (Saunders et
al., 2016:136)
Applicability to this study
Ontology Nature of reality or being ‘Reality’ is the practical
consequences of ideas: overall
this study accepts that there
may be more than one solution
(or reality) to the research
problem, but in the end the
practical framework becomes
the reality of the research.
Epistemology What constitutes acceptable
knowledge
Focus on problems,
practices and relevance: the
subjectivism, mixed methods
and abductive approach of this
study are all accepted under
pragmatism, since the ultimate
aim is to have a practical
answer to the research
question.
Axiology Role of values in research Researcher’s values drive the
reflexive process of inquiry:
this researcher’s
interpretations and subjective
approach are thus allowed.
Source: Researcher (adapted from Saunders et al., 2016:136)
3.3 APPROACH TO THEORY DEVELOPMENT
The next layer of the “research onion” (Figure 3-1, page 49 above) comprises of the
researcher’s approach to theory development. It is important that the researcher considers
this step so that the different parts of the research process, i.e. the paradigm, assumptions
and the research design, logically align, which in turn raises the credibility and validity of the
56
study. According to Trochim and Donnelly (2008:16), there are two broad methods known as
deductive and inductive approaches, which are utilised during the process of logical reasoning.
Deductive reasoning works from the general to the more specific (Trochim & Donnelly,
2008:17). Using a deductive approach thus entails that one’s research starts with theory, often
developed from reading of the academic literature (Saunders et al., 2016:145). The theory is
then typically narrowed down to some specific hypotheses for which one then designs a
research strategy to test it on (Trochim & Donnelly, 2008:17; Saunders et al., 2016:145).
Burney (2008:4) agrees and adds that the deductive approach is informally known as the "top-
down" approach, which can be remembered graphically as a “waterfall” (see Figure 3-2, page
56 below). He notes that with a deductive approach one’s conclusions follows logically from
the available facts (premises) (Burney, 2008:4). Arguments based on laws, rules and accepted
principles are generally used for deductive reasoning (Burney, 2008:7).
Conversely, inductive reasoning works the other way, moving from the specific observations
to broader generalisations and theories (Trochim & Donnelly, 2008:17). Thus, if your research
starts by collecting data to explore a phenomenon and you generate or build a theory (often
in the form of a conceptual framework), then you are using an inductive approach (Saunders
et al., 2016:145). As further discussed below and based on the research objectives, it can be
seen that the inductive approach is applicable to this study since a phenomenon (assessments
tasks, cognitive demands and SOLO levels) is explored so as to build a conceptual framework.
Burney (2008:5) agrees with the above definition of inductive reasoning and adds that the
inductive approach is informally known as the "bottom up" approach, which can be
remembered graphically as “hill climbing” (see Figure 3-2). He notes that observations tend to
be used for inductive arguments (Burney, 2008:7). Trochim and Donnelly (2008:17-18)
furthermore state that inductive reasoning, by its nature, is more open-ended and exploratory,
especially at the beginning of the implementation of the research methods.
57
Figure 3-2: A schematic representation of deductive and inductive reasoning
Source: Burney (2008:6)
The above two main approaches to theory development can however be combined (Creswell,
2014:186) by moving back and forth between data and theory (or theory and data). This
process is known as abductive reasoning (Saunders et al., 2016:149). Saunders et al.
(2016:148) note that abductive reasoning is typically used in practice by many business and
management researchers. These researchers use data to explore a phenomenon, identify
themes and explain patterns, then generate a new (or modify an existing) theory, which is
subsequently tested, often through additional data collection (Saunders et al., 2016:152).
Saunders et al. (2016:149) also explain the following about abductive reasoning: “in our
experience [abductive reasoning], [is] often advantageous to do so, although often one
approach or another is dominant”. The dominant approach depends on the emphasis of the
research and the nature of the research topic (Saunders et al., 2016:149).
In line with the above and as motivated further below, the dominant approach for this study
will be an inductive approach with a secondary abductive approach. The main reason for this
is that observations (the studying of the financial accounting exams) will be transformed
through analysis (which includes interpretation of the assessment tasks in terms of the SOLO
taxonomy) into more general themes. At the same time, a small but material, secondary
deductive approach will also be followed, where the themes will be considered against
different exams in order to test applicability and to test whether further evidence which
supports the theme exists. This secondary process has the benefit of raising the study’s
58
validity and credibility. This abductive approach fits with a pragmatic paradigm which frames
this study.
Saunders et al. (2016:149) warn that abductive and, particularly, inductive research can be
much more protracted, since the ideas (themes) tend to be based on a much longer period of
data collection and analysis, and have to emerge gradually. To mitigate this risk, the topic of
the study was aligned with this researchers’ career experience. This researcher has spent
many years as a financial accounting student, accountant, lecturer and scholar, and therefore
can draw on this experience during the data analysis phase. In essence this researcher has
informally analysed many financial accounting exams throughout the years.
In conclusion, the approach used for this study is thus mainly and primarily inductive in nature,
but will include some abductive reasoning during the process.
3.4 RESEARCH DESIGN
From the literature on research methodology, it is seen that different authors follow different
routes on suggesting the systematic process which is to be followed in research, but in the
end it is clear that the research design type, corresponding methodology and research
methods should at least be reflected on and must be well motivated (Saunders et al.,
2016:164).
According to Saunders et al. (2016:163), a research design is the general plan of how one will
go about answering the research questions. Henning (2005:39) agrees that the design type
or genre is what gives coherence to the plan of a study. The design should include detail about
data sources and analysis (Saunders et al., 2016:163). Bertram and Christiansen (2017:40)
argue that the research design is an important plan, but that it is not a fixed plan that proceeds
in a very structured way. Bertram and Christiansen (2017:40) also see the research design as
a flexible and non-linear process that is influenced by practical considerations.
In the “research onion” (Figure 3-1, page 49 above) it can be seen that Saunders et al. (2016)
structure the research design into the “methodological choice” followed by the “research
strategy”, “time horizon” and “techniques and procedures”. The “techniques and procedures”
is the label that refers to data collection and analysis (or research methods).
In order to further focus one’s mind over the issue of research design, it is also helpful to
consider the purpose of one’s research design. Research can be designed to fulfil either a
descriptive, exploratory, explanatory or evaluative purpose, or some combination of these
(Saunders et al., 2016:174). The novelty of this study’s research questions (paragraph 1.2) is
59
exploratory in nature, but the fact that the study aims to address a specific research problem
in a practical way brings in elements of a descriptive as well as an explanatory study.
Therefore, this study has a combined purpose.
3.4.1 Methodological choice
It is evident that the nature of data for research purposes can be numeric (quantitative data)
or non-numeric (e.g. text, images, sound bites, etc.) which is qualitative data (Trochim &
Donnelly, 2008:142). According to Saunders et al. (2016:164) the first methodological choice
is whether one follows a quantitative, qualitative or mixed methods research design.
A narrow view of the methodological choice is therefore one of qualitative data collection and
analysis or quantitative data collection and analysis (Creswell, 2014:4; Saunders et al.,
2016:165). Saunders et al. (2016:165) also explain that many research designs are likely to
combine quantitative and qualitative elements. A quantitative research design may, as part of
addressing the research problem, also require respondents to answer some open-ended
questions (which will lead to qualitative data) or some qualitative research data may be
analysed quantitatively (Saunders et al., 2016:165). It thus makes sense that Saunders et al.
(2016:165) view quantitative and qualitative research as two ends of a continuum, which in
practice are often mixed. This dualistic approach is therefore known as a mixed methods
approach (Creswell, 2014:3) and is further discussed below.
Creswell (2014:4) and Saunders et al. (2016:166) are however in agreement that a more
complete (broader) way to assess this methodological choice is to interpret quantitative and
qualitative methodologies through their associations to the philosophical assumptions,
research approaches and research strategies. Some authors may even consider and discuss
quantitative, qualitative and mixed methods research as a separate paradigm (Creswell,
2015:2; De Vos et al., 2018:222), but for this study Saunders et al.’s (2016:164) view of one’s
choice to follow a quantitative, qualitative or mixed methods research design as being a
methodological choice, is followed.
Accordingly, a broader discussion of quantitative research involves that it is described as an
approach for testing objective theories examining measurable variables (Creswell, 2014:4).
These variables are typically measured on instruments that lead to numeric data which can
be statistically analysed and generalised to the larger population (Creswell, 2014:4). The
primary role of quantitative research is to test a hypothesis, which is defined as a proposition
regarding the relationship between two or more variables (Struwig & Stead, 2001:4).
Correspondingly, quantitative research is typically deductive in nature (Struwig & Stead,
60
2001:5). Quantitative research is therefore generally associated with positivism, especially
when used with predetermined and highly structured data collection techniques (Saunders et
al., 2016:166). It is, however, important to realise that there may be instances where
quantitative research may draw on an interpretivist philosophy (certain kinds of surveys for
example), or where quantitative research may draw on a realist or pragmatism paradigm as
part of mixed methods research (Saunders et al., 2016:166). Other typical elements of
quantitative research are that it seeks to prove or disprove causality (i.e. cause and effect
relationships) (Struwig & Stead, 2001:5) and seeks to set up replication (i.e. a research
process that other researchers can exactly follow to replicate a study and in so doing validate
the original findings and results) (Struwig & Stead, 2001:6). Quantitative research is also
typically linked to the natural sciences (Struwig & Stead, 2001:6). Sample sizes for quantitative
research need to be sufficient in order for statistical analysis to be executable (Bertram &
Christiansen, 2017:63).
In terms of this study, it is explained below (where mixed methods research is discussed) why
and how quantitative research applies to this study.
Creswell (2014:4) describes qualitative research as “an approach for exploring and
understanding the meaning individuals or groups ascribe to a social or human problem”.
Saunders et al.’s (2016:168) angle of incidence, in agreement with Creswell (2014), is that
qualitative research is interpretive in nature, “because researchers need to make sense of the
subjective and socially constructed meanings expressed about the phenomenon being
studied”. Struwig and Stead (2001:11) note that the term “qualitative research” does not
describe a single research method, but rather that there are many research methods
associated with qualitative research. Struwig and Stead (2001:11) add that qualitative
research is found in many different fields, such as management science, psychology,
education, and history. In light of the aforementioned view of Struwig and Stead (2011:11), it
makes logical sense that many authors on research methodology focus on the typical
characteristics of qualitative research (as compared to trying to formulate an exhaustive
definition), since qualitative research as a concept can “mean many things” (Struwig & Stead,
2001:12) and encompasses multiple possibilities for research designs (Henning, 2005:39).
The purpose of Table 3-4 (page 61) is to illustrate the typical characteristics of qualitative
research as discussed by authoritative scholars. Table 3-4 also shows how these
characteristics are applicable to, and applied in, this study. Table 3-4 is not meant to be an
exhaustive or exclusive list.
61
Table 3-4: Typical characteristics of qualitative research as applied in this study
Typical characteristics
of qualitative research
Source Applicably of said characteristic to
this study
Exploring and
understanding the
meaning an individual or
group ascribe to a social
or human problem.
Creswell (2014:4)
Henning (2005:3)
Saunders et al. (2016:168)
Struwig and Stead (2001:17)
This study does not aim to observe
participants directly in a naturalistic
setting as can be expected in many
qualitative studies (according to
Saunders et al., 2016:168). In Table
3-2 (page 52) it is explained that an
indirect process of studying the
meaning that participants (students)
may ascribe to their “problem” (the
assessment tasks) will be followed,
since the researcher will draw on his
own experience as an accountant,
lecturer and former Financial
Accounting 1 student to place himself
in the students’ “shoes” when they
write the exam, so as to obtain a richer
understanding of how the assessment
tasks and the SOLO taxonomy relate in
ultimately setting an observable level
or standard of the assessment.
Inductive reasoning,
subjective involvement of
the researcher and
drawing on an
interpretivist philosophy.
Creswell (2014:183)
Henning (2005:16)
Saunders et al. (2016:168)
Struwig and Stead (2001:16)
All of these characteristics are
applicable as discussed above.
Many research strategies
are provided for, some of
the main ones being:
action research, case
study
research, ethnography,
grounded theory,
naturalistic enquiry,
phenomenology, archival
and documentary
research etc.
Creswell (2014:14)
Henning (2005:49)
Saunders et al. (2016:178)
Struwig and Stead (2001:13)
One or more of these strategies
(simply referred to as “design types” by
some authors, see paragraph 3.4.2
below), will be drawn upon and applied
in paragraph 3.4.2 below.
Source: Researcher
62
Creswell (2014:4) describes mixed methods research as involving the “combining or
integration of qualitative and quantitative research and data in a research study”. Saunders et
al. (2016:171) agree with this view and add that it is possible that quantitative and qualitative
methods can be mixed at different (or all of the) stages of one’s research.
In is acknowledged that this study is primarily qualitative in nature. The data collected, for
example, mainly consist of written text in the form of Financial Accounting 1 exams. However,
the main reason that this study is considered mixed methods is that the financial accounting
exams also state the number of marks and time allowed as numeric (quantitative) data which,
although vastly less in scope compared to the textual data (qualitative data), is still useful and
entails that this study is one of mixed methods research.
A secondary but important consideration is that during the analytical phase of this study, the
technique of “content analysis” will form a small but important part of addressing the research
questions. Saunders et al. (2016:608) succinctly describe content analysis as an analytical
technique that codes and categorises qualitative data in order to analyse them quantitatively.
The central tenet behind content analysis, in terms of documents, is that words or other images
are transformed from a qualitative form into a quantitative systematic form (De Vos et al.,
2018:381). The benefit of this process, amongst others, is that it allows the researcher to count
frequencies and examine relationships between variables in the data (Saunders et al.,
2016:612). It is the nature of financial accounting exams as a document to, for example,
include issues such as number of assessment tasks and number of words of reading material.
It was therefore obvious at the onset of this study that content analysis will be a feature. So,
although most scholars seemingly point to content analysis as a feature of qualitative research
(see for example Henning, 2005:49; Saunders et al., 2016:608; Mouton, 2013:105; Bowen,
2009:32; De Vos et al., 2018:380), there is an argument to be made that content analysis is a
separate quantitative method due to the utilising of numeric data in a process of statistical
analysis. This argument is for example supported by the view of Saunders et al. (2016:178)
when they refer to documentary research as a design that involves quantitative or qualitative
research, or a mixed design combining both. Mouton (2013:166) also suggests that content
analysis, in terms of descriptive statistics, can be a quantitative method. Neuman (cited by
Struwig & Stead, 2001:14) states that content analysis can be quantitative or qualitative
evaluations of text. For this study, both views are accommodated since a quantitative design
is in any event included, and since a pragmatic approach with the main focus of addressing
the research objectives is followed. One can however state that a secondary reason of why
this study is therefore considered a mixed methods study is the fact that during the analytical
phase, content analysis as a form of a quantitative technique will also be applied.
63
The above approach of the qualitative research part of the study being much broader in scope
than the quantitative research part is in line with Saunders et al.’s (2016:172) view that under
mixed methods research the different approaches can be utilised in an unequal way so that
one methodology has a dominant role, while the other plays a supporting role as determined
by the purpose of the research project.
In Saunders et al. (2016:172 and 2016:613), it can also be seen twice that researchers are
warned that an excessive process of ‘quantitizing’ qualitative data may lead to loss of
exploratory or explanatory richness (which in turn may lead to the research objectives not
being achieved). This warning was headed by this researcher and it is therefore emphasised
in this chapter that the primary methodological approach is qualitative in nature, as
underpinned by an interpretative paradigm and inductive approach, but in order to
accommodate the limited but material quantitative data in the financial accounting exams as
well as utilise the valuable technique of content analysis and ultimately answer the research
question to the fullest, the philosophy of pragmatism and a mixed methods design are drawn
upon. Saunders et al. (2016:170), who utilised the views of Tashakkori and Teddlie (2010)
and Nastasi et al. (2010), submit that for pragmatists, the nature of the research question, the
research context and likely research consequences are driving forces determining the most
appropriate methodological choice. In terms of this study this aforementioned statement is
appropriate.
The methodological choice in terms of this study is therefore in summary:
i) a basic or simple mixed methods research approach (as listed in Creswell, 2015:6)
since only a small part of the data analysis phase of the study will utilise basic
quantitative statistics. The overall design is a convergent design (Creswell, 2015:6)
since (although the order of magnitude differs) both qualitative and quantitative
data are collected and the results from analysis from both sets of data will be
merged in order to address the research questions as comprehensively as
possible, and in so doing raising the validity and credibility of the study.
ii) a fully integrated mixed methods research design (Saunders et al., 2016:170),
since although the prominence and extent of the quantitative part of this research
is subordinate to the qualitative part of this research, both methods are used at
every stage of the research.
Table 3-5 (page 64) summarises the reasons (other than the nature of the data) for this study
utilising a mixed methods design.
64
Table 3-5: Reasons for using a mixed methods design
Reason (Saunders et al., 2016:173) Applied to this study
Complementarity The quantitative and quantified data will allow the
meanings and findings of the primary qualitative
part of the study to be elaborated, enhanced,
clarified, confirmed, illustrated or linked.
Interpretation The quantitative data and analysis will be used to
help to explain relationships between variables, i.e.
the assessment tasks and the SOLO taxonomy as
well as other themes that will be emerging from the
primary qualitative research design.
Credibility Even though limited in scope, the nature of
quantitative data in terms of financial accounting
exams (i.e. time per assessment question and the
number of marks) as well as the results of content
analysis, bring a bit of balance on the scale of
objectivism vs subjectivism. Especially under a
pragmatic approach, it is appropriate to have the
“numbers” serve to support the qualitative analysis.
This is mainly done by the statistics being
transparent to follow. This has the effect of raising
the study’s credibility.
Source: Researcher (adapted from Saunders et al., 2016:173)
3.4.2 Research strategy
Saunders et al. (2016:164) view the research strategy as part of the broader research design
as they illustrate in the “pink-shaded” part of the “research onion” (Figure 3-1, page 49).
Saunders et al. (2016:164) also state that within one’s research design, one will need to use
one or more research strategies to ensure coherence within your research project. The
research strategy is further explained as “the methodological link between your philosophy
and subsequent choice of methods to collect and analyse data” (Saunders et al., 2016:177).
Many authors refer to this research strategy as the research design type (see for example:
Henning, 2005:49; Mouton, 2013:57; and De Vos et al., 2018:312). In taking a pragmatic
approach, the words “research strategy” and “research design type” is used as synonyms and
interchangeably for the purposes of this study.
Before the design type of this study is thoroughly motivated, it is worthwhile to note for the
purpose of credibility that the research methodology sources of this study, besides being
65
“stars” of scholarship themselves, use the work of other well-known authors intensively. A
small selection of examples include Saunders et al. (2016:178), who refer to the work of
Denzin and Lincoln (2011) or De Vos et al. (2018:380), who refer to the work of Babbie and
Mouton (2001) as well as Henning (2004). Creswell (2014:14), for example, builds on the work
of Moutsakas (1994) and Corbin and Strauss (2007).
In paragraph 3.4.1 it was explained that the methodological choice of this study is mixed
methods research. The quantitative design type will now be briefly discussed to be followed
by a broader discussion and interrogation of the qualitative design type.
The quantitative design type for this study is driven by three types of quantitative data. These
are: i) the number of marks that each exam question counts; ii) the time allowed per
assessment task; and iii) the quantitative data derived from the content analysis process which
can include the amount of time per assessment task or the number of pages of reading
material or other frequencies that will be discovered. The aim is to utilise this limited but
potentially influential quantitative data to support the qualitative themes emanating. Therefore,
basic statistics will be used. The design type to be used is one of basic descriptive statistics,
the purpose of which is to enable one to describe (and compare) variables numerically
(Saunders et al., 2016:527). The approach used for this study will, in line with Saunders et al.,
(2016:527), focus on the two aspects of description, namely i) the central tendency and ii) the
dispersion.
Central tendency includes statistics such as the mode (the value that occurs most frequently),
the median (middle value), the mean (or average) and variations (reasoned custom
adjustments) of these (Saunders et al., 2016:528). The dispersion includes statistics such as
range (highest and lowest values), deciles or percentiles (ranking of data), and variance (such
as the standard deviation or coefficient of variation) (Saunders et al., 2016:528). For this study,
the statistics will be allowed to naturally manifest as driven by the research questions and the
data (in line with Saunders et al., 2016:527), but will be illustrated in a transparent and logical
way so as to maintain credibility.
The broader discussion and interrogation of the qualitative design type now follows.
Authors are in agreement that, especially in terms of qualitative research, there is a “baffling”
choice of design (De Vos et al., 2018:312). Henning (2005:39) acknowledges that some
methodologists are coming up with new design types and that the possibilities for hybridisation
and finding a descriptor for a design type are limitless. Tesh (1990:58) already listed 26
possible qualitative designs in 1990. Henning (2005:39) and Saunders et al. (2016:177)
66
therefore agree in recommending that a design type should at least explore its roots which are
likely to be found in the commonly or traditionally accepted design types. De Vos et al.
(2018:323) emphasise that qualitative research designs often overlap and that researchers
should be wary of deciding on a design as an “all purpose vehicle suitable for any type of
road”, but state that the design should rather be led by the research problem. De Vos et al.
(2018:323) are seemingly advocating for a measure of pragmatism here. Nonetheless, the
point is that a researcher should be wary of “mechanically” going “through the motions” without
applying his mind.
Therefore, although it is clear from this study’s research objectives and data that the
document analysis design type would be appropriate, the case study design type is also
briefly discussed in order to demonstrate the consideration of a possible relevant alternative.
3.4.2.1 Case studies
According to De Vos et al. (2018:320), a case study design is rather a choice of what to study
than being a pure methodological choice, since this design can easily adapt to the other
methodological frameworks such as phenomenology, grounded theory or ethnographic
research. The core feature of a case study design, however, is that the researcher develops
an in-depth analysis of a case in its context (Bertram & Christiansen, 2017:42). The “case”
being studied may refer to a program, event, activity, process, one or more individuals
(Creswell, 2014:14), or even a period of time (De Vos et al., 2018:321). A case is normally
bounded by time, context or place (Creswell, 2014:14), which implies one of the reasons that
this type of research has very limited generalisability (another reason being that case studies
usually has an in-depth focus on a limited amount of cases or even just one case).
Nonetheless, a case study design may for example typically prove valuable when the objective
of the research is to understand the meaning participants give to their life experience, since
the researcher can become immersed in their social worlds and in doing so find patterns in
their lives, words, actions or contexts (De Vos et al., 2018:320).
Case study research typically involves the collection of detailed data about a case by using
various methods (Yin as cited by Creswell, 2014:14) including interviews, documents,
observations or archival records (De Vos et al., 2018:321).
For this study there is an argument to be made that the financial accounting examinations and
their questions (assessment tasks) present cases to be studied. The primary focus of the study
as represented by the research objectives is however to analyse the examinations with and
against the SOLO taxonomy to crystallise themes. The aim for example, does not include
67
describing a full in-depth context around the examiners whilst setting their papers (it is not this
study’s aim to describe what makes the “examiners tick”). So, although elements of case study
research will be present in this study, the case-study design is not the most fitting for this
study.
3.4.2.2 Document analysis
As the name suggests, the primary feature of this design type is that documents will be used
as the research data that are to be analysed. A document is a durable repository that can
contain textual as well as visual, and interestingly if in electronic format, audio data (Lee as
cited by Saunders et al., 2016:183). In paragraph 1.4.2 it was already stated that the document
analyses design will be utilised for the qualitative part of this study. The reason given is that it
can be seen from the research objectives that the Financial Accounting 1 summative
examinations, as a form of document that comprises mainly qualitative (textual) data, will be
analysed from the data against the SOLO taxonomy, and from the SOLO taxonomy against
the data in an abductive interpretive process. Due to this design type’s importance for this
study it is appropriate that a credible form of a definition is given for the design type of
“document analysis”. All the scholars mentioned in this section above are in agreement as to
the characteristics of document analysis, but due to document analysis being the main and
sole topic of Bowen’s (2009:27) work, the definition below was paraphrased form his work:
Document analysis is thus a systematic procedure for reviewing or evaluating documents
that requires that data be examined and interpreted in order to elicit meaning, gain
understanding, and develop empirical knowledge (Bowen, 2009:27).
Saunders et al. (2016:183) describe this design type as “[a]rchival and documentary research”
and point out that especially since the era of digitalisation, the range of archival and
documentary sources that are potentially available is exceptionally vast. A few examples
mentioned by Saunders et al. (2016:183) include email, diaries, minutes of meetings,
government statistics and media articles. Smith (2011:143) on the other hand refers to this
design type as “[a]rchival research” with the term “archive” being used in a “deliberately broad
approach” to include, amongst others, historical documents, texts, journal articles, company
annual reports, company disclosure and many more. Bowen (2009:27) refers to the design
type as “[d]ocument analysis”, while De Vos et al. (2018:376) use the term “[d]ocument study
and secondary analysis”. De Vos et al. (2018:377) even mention suicide notes as a form of
document to be studied. What the aforementioned show is that although terminology may
differ slightly, the design type is well recognised by leading scholars.
68
Document analysis is an iterative (“building”) process of developing findings, and typically
combines one or more of the data analysis techniques given in Table 3-6 (page 68). It can be
seen that these techniques are not necessarily mutually exclusive and may overlap. The
techniques applicable to this study are elaborated upon in paragraph 3.4.4 below.
Table 3-6: Typical techniques for analysing documents
Technique Brief description Source
Thematic analysis The search for and recognition of patterns within
the data. These patterns form themes that become
the categories for analysis (Bowen 2009:32).
Braun and Clarke (cited by Saunders et al.,
2016:579) refer to Thematic Analysis as a
“foundational method for qualitative analysis”.
Bowen (2009:32)
Braun and Clarke
(cited by Saunders
et al., 2016:579)
Content analysis Authors have different views on content analysis
(as discussed in paragraph 3.4.1).
The view used for this study is that content analysis
is an analytical technique which codes and
categorises qualitative data in order to analyse
them quantitatively.
Saunders et al.
(2016:608)
Textual analysis The analysis of text in order to understand and
interpret the deeper meaning the document may
have. Mouton (2013:167) also groups
hermeneutics and textual criticism as special forms
of textual analysis.
Mouton (2013:167)
Semiology Identifies words and especially images as signs
that offer complex meaning or significance beyond
the surface of the text.
De Vos et al.
(2018:381)
Linguistic analysis Exploring the use and meaning of words and
phrases in a document. Essentially includes what
meaning can be derived from the nature of the
language, for example issues around syntax and
semantics.
De Vos et al.
(2018:381)
Source: Researcher
It should also be noted that for this study the financial accounting examinations are considered
secondary documents, since they were not generated by this researcher himself and was
originally created for a different purpose than this research. The data analysis process is
however a primary process, since it will not entail the reworking of already analysed data,
which is thus not a “secondary data analysis” design but the use of secondary sources as part
69
of document analysis. This view is supported by Smith (2011:143), Saunders et al. (2016:184)
and De Vos et al. (2018:383).
The various scholars also emphasise that the limitations, advantages and disadvantages of
document analysis should be carefully considered, and they indeed go on to do just that in
their writings. In Table 3-7 the primary advantages and disadvantages (with limitations or risks
being grouped under disadvantages) are reflected upon in terms of this study.
Table 3-7: Reflecting on the advantages and disadvantages of document analysis
Advantage Discussion
Efficient (Bowen, 2009:31) Time is saved due to data in essence being selected and
not collected for this study. This refers to the financial
accounting exams already being in existence and not
needing to be generated. Bowen (2009:31) also states that
document analysis is less time-consuming and therefore
more efficient than other research methods. For example,
documents do not need to be transcribed like interviews. In
this researcher’s view this statement is possibly even more
accurate if the researcher has experience in the research
topic (which is indeed the case for this study).
Low cost (De Vos et al.,
2018:382).
The documents obtained for this study (being the financial
accounting examinations) were given voluntarily and at no
cost. There was also no need to travel, etc.
Lack of obtrusiveness and
reactivity (Bowen, 2009:31)
The financial accounting examinations stay what they are
(i.e. are non-reactive) and are not changed by or during the
research process. This aids the credibility and validity of the
study. Studying documents means participants are not
perturbed, which have a range of benefits including limited
ethical concerns, availability at all times, and digitalisation.
Humans may react to a myriad of factors at any specific
time, while the document stays what it is. Document
analysis is also the design this researcher (as an introvert)
feels most comfortable with (which is acceptable under a
pragmatic paradigm according to Morgan (2014:7)).
Disadvantage/limitation/risk Discussion
Not fit for purpose (Saunders et
al., 2016:184)
The original purpose of a document likely had nothing to do
with one’s research, therefore researchers must be
sensitive of the effect this has on data analysis and
interpretation. The document may for example end up as
70
not having sufficient or appropriate data to answer the
research questions. During the planning phase of this
study, a few financial accounting examinations were
reviewed to ensure that these risks did not materialise.
Biased selectivity (Bowen,
2009:32)
The collection (population) of documents may be
incomplete or the documents may be influenced by the
originators’ bias. For this study this is not seen as a material
risk, since the creators of the financial accounting
examinations had no incentive or opportunity to try an
influence of this study. The methodology of this study, as
framed in an interpretive and pragmatic paradigm, provides
that the data “should speak for itself” and does not rely on
predefined findings.
Lack of standard format (De Vos
et al., 2018:383).
A lack of a standard format or an unreadable electronic
format may hamper the data analysis process significantly.
For this study the financial examinations were requested
and received electronically and if problems are
encountered a solution will be found.
Source: Researcher
3.4.3 Time horizon
Grey (2014:35) and Saunders et al. (2016:200) are in agreement that the researcher must
consider the timeframes or timescales of their projects. The two types of timeframes wherein
a study is placed are stated as cross-sectional or longitudinal (Grey, 2014:35). A cross-
sectional study is short-term in nature, utilising a ‘snapshot’ approach where the data are
collected at one point in time (Grey, 2014:35). A longitudinal study can be seen as similar to
a dairy or a series of snapshots and therefore represents events over a given period (Saunders
et al., 2016:200). The timeframe of a study will depend on the research question (Saunders et
al., 2016:200). Grey (2014:35) and Saunders et al. (2016:200) in also in agreement that most
research studies are cross-sectional, mainly because of the pressure of time and resources.
The aforementioned should not detract from the value of longitudinal research, which is its
capacity to study change and development (Saunders et al., 2016:200).
In terms of this study the financial accounting 1 summative examinations, as a form of
document that comprises mainly of qualitative data, will be analysed from the data against the
SOLO taxonomy and from the SOLO taxonomy against the data in an abductive interpretive
process (as planned in paragraph 1.3 and 1.4). These financial accounting 1 examinations
represent a “snapshot” in time and are not a series of documents over time (which can be a
71
topic of another study). This study is thus a cross-sectional study. In paragraph 3.4.4.1 it is
explained that this fact should not reduce the validity of this study.
3.4.4 Data collection and analysis (research methods)
The final part and inner core of Saunders et al.’s (2016:124) “research onion” (Figure 3-1,
page 49) comprises the data collection and analysis techniques which are also known as the
research methods. In considering the research methods that will be used with the document
analysis strategy of this study, De Vos et al. (2018:381) provide a useful list of practical steps
concerning the execution of a document analysis. These steps are adapted in Table 3-8 below
as a starting point for the discussion of the research methods to be used.
Table 3-8: Practical steps of document analysis
Steps Applied in this study
Formulation of the research questions. This is done in paragraph 1.2.
Keep a research diary of your courses of
action and analytic thoughts over the
timeframe in which the research is done.
One benefit of this process is that it brings
efficiency. An example of this was the utilisation of
“dead time” which was used by this researcher to
reflect or read about or analyse parts or all of this
study, for example while waiting in a que at a bank.
This saved time later on.
Find sources of material and generate and
archive.
The data of this study comprises summative
financial accounting examinations. These
documents make up the target population, which
together with the sample method and sample size
is discussed in paragraph 1.4.2 as part of the
planning phase of this study, but is also mentioned
below for ease of reference.
Transcribe the text in some detail. As noted by De Vos et al. (2018:381), this step is
not necessary for existing documents. The financial
accounting examinations of this study is an
example of existing documents.
Critically read and re-read the documents. The effects of this step are documented in Chapter
4 below.
Coding of the data. The process of coding will be discussed in this
section below.
Analyse the data This study will use content analysis, thematic
analysis and textual analysis to analyse the
financial accounting exams. These techniques,
72
including their corresponding processes of
interpretation, will be further discussed below.
Consider the credibility, validity and reliability
of the study and your findings.
This is done throughout the study, but a summary
will be provided in this chapter below.
Write up your findings. This is done in Chapters 4 and 5 below.
Source: Researcher (adapted form De Vos et al., 2018:381)
3.4.4.1 Target population and sampling frame
In the light of the research objectives, the target population comprises the following:
Final summative financial accounting examinations pitched at NQF level 5 as part of a financial
accounting qualification which may eventually lead to any professional accounting
designation. This is a typical first year accounting module, also known as “Financial
Accounting 1”. A review of the Bachelor’s degrees registered with SAQA as specialising in
financial accounting confirms the aforementioned (SAQA, 2017).
These summative examinations are set as those from the 26 public universities (USAf, 2020)
in South Africa, since i) the researcher’s experience stems from being employed at a public
university, currently being at the North-West University, and ii) the researcher is aiming the
study’s contribution toward the public universities.
Based on discussions with the custodians (the universities) of these examinations, it was
noted that most of them felt more comfortable to provide documents from a year prior to this
study, primarily since the documents would then already be in the public domain and no
accusations of “data leaks” could be made against them. They also felt that they could then
provide moderated and “tested” documents which better reflect their assessment practices. In
line with the ethical approach discussed in paragraph 1.5, this researcher will therefore accept
final Financial Accounting 1 examinations from the year 2016 and from one university who
was only prepared to provide 2015’s exams. Later on, one university sent a 2018 examination
which was also accepted. The differences between the examinations’ ages are not a concern
for this study and should not be seen to reduce the validity of this study, since experience
dictates that foundational financial accounting as a subject stays relatively unchanged in its
core concepts (e.g. dealing with debtors, creditors, bank and cash, fixed assets, etc.) and that
the findings of this study in terms of the data set (examinations) and the SOLO taxonomy will
be transferable even to newer data sets, because the primary question is one of “how to” apply
the SOLO taxonomy in order to evaluate the level of assessment of a Financial Accounting 1
exam.
73
3.4.4.2 Sample method
The sample method for this study is firstly categorised as non-probability sampling (Cohen et
al., 2011:155). The sample is considered non-probability, since per the research objectives
and design there is no objective which aims at statistical generalisability. It is important to note
that as is expected from a qualitative inductive approach, inferences and generalisations in
terms of broader themes can still be made (Onwuegbuzie & Leeach as cited by Cohen et al.,
2011:161).
Cohen et al. (2011:155-156) discuss three types of non-probability sampling, i.e. convenience
sampling, quota sampling and purposive sampling. For this study the applicable method
chosen and discussed below is “purposive sampling”.
Purposive sampling is stated as often being a feature of qualitative research and involving the
researcher hand-picking cases to be included in the sample, based on the researcher’s
judgement of the cases’ typicality or characteristics being sought (Cohen et al., 2011:156).
Purposive sampling is typically used to build a sample which meet specific needs (Cohen et
al., 2011:156).
From the study’s research objectives as well as the target population it can be seen that a
sample of summative financial accounting examinations pitched at NQF level 5 from public
universities is needed. The aforementioned, together with the following reasons, constitute
why purposive sampling is appropriate.
For this study it was decided (to enhance credibility and applicability) to build in a form of
representativeness by purposively selecting some of the universities with larger student
numbers, as well as universities from different provinces and different types (distance,
technology and traditional contact universities). Purposive sampling lends itself to the
attainment of a good measure of representativeness (Teddlie & Yu cited by Cohen et al.,
2011:156) which Cohen et al. (2011:157) also refer to as “typical case sampling”. Teddlie and
Yu (cited in Cohen et al., 2011:156) note that purposive sampling provides for greater depth
of study as compared to probability sampling (although breadth of study is traded off).
However, for the purposes of achieving this study’s objectives in light of a primarily qualitative
paradigm, depth of study is what is sought, and hence purposive sampling is apt.
This study also incorporates a small degree of “convenience sampling”, which is a form of
non-probability sampling where the researcher samples from those to whom he has easy
access (Cohen et al., 2011:156). For this study the sample of universities whose examinations
will be analysed will be based on the universities to which the researcher has access (i.e.
74
universities willing to assist). It is submitted that in light of the research design, objectives and
paradigm that this should not affect the credibility of the study as long as ethical aspects are
adhered to (e.g. interpreting without preconceived bias and not changing the received exams).
3.4.4.3 Sample size
Onwuegbuzie and Leeach (cited by Cohen et al., 2011:162) set two main recommendations
for a sample of qualitative data which are applicable to this study:
1. The sample should be large enough so that “thick” or “rich” descriptions can be made.
“Thick description” is a technique of providing enough detail around the research so that
the reader can judge whether the findings can be relevant (i.e. transferable, which is not
to be confused to wider generalisability) to another context (Scott as cited by Bertram &
Christiansen, 2017:123). The aim is one of more credible and useful findings stemming
from achieving the research objectives.
2. Data saturation is usually indicative of a sufficient sample. Saturation means that one can
stop collecting data when the fresh data no longer reveals new properties or insights
(Creswell, 2014:189).
Based on the researcher’s experience as well as a preliminary review of examinations, it is
noted that:
Financial Accounting 1 examinations contain more than one question (assessment
task) and is quite descriptive, similar to being mini-case studies.
The examinations also almost universally assess the same fundamental topics (such
as debtors, creditors, bank and cash, fixed assets). This indicates a point of data
saturation in the sense of accounting topics.
In light of the above, financial accounting summative assessment (exams) of seven
universities are to be analysed. These universities are listed below in no particular order and
do not correspond to the labels that will be applied in Chapter 4. For selection criteria, please
refer to the sample method above. The universities are:
1. Central University of Technology
2. North-West University
3. University of the Free State
4. University of Johannesburg
5. University of South Africa
6. University of the Witwatersrand
7. University of Zululand
75
This sample size will be sufficient to achieve the research objectives.
3.4.4.4 The process of coding
Coding is the process whereby sections of qualitative data are categorised and labelled in
order for patterns to emerge (De Vos et al., 2018:411). Coding is a primary step for the analysis
of data under any qualitative research design (De Vos et al., 2018:410) while also being the
foundation for thematic analysis (Saunders et al., 2016:579). Coding is a way to manage and
find meaning from (the typically large amount of) qualitative data (Saunders et al., 2016:580).
Coding is a relatively simple but versatile and valuable tool for addressing a research question
(Saunders et al., 2016:582).
Saunders et al. (2016:580) succinctly explain a “code” as a short phrase that may also be a
single word or abbreviation. An extract of coded data is known as a unit of data and typically
contain a number of words, sentences, paragraphs or even visual images (Saunders et al.,
2016:580). A unit of data may in the end be coded with more than one code or may overlap
under different codes (Saunders et al., 2016:580).
There are three types of codes that are commonly used, and these codes are either derived
for the data or existing theory or literature. Figure 3-3 visibly shows these types of codes:
Figure 3-3: Sources and types of codes
Source: Saunders et al. (2016:583)
Saunders et al. (2016:580) and De Vos et al. (2018:411) remind their readers that coding (and
indeed the bigger process of qualitative data analysis) will likely not occur as a simple linear
progressive process. The main reason for this is that as one codes, new understandings
emerge from the data, which may alter the codes themselves or the relationships between the
76
different pieces of data. Data are thus analysed and re-analysed until the research questions
are answered.
The process of coding as envisaged for this study (although in the light of a pragmatic
approach and the thematic discoveries, some adaptations may occur) will now be described
in the following paragraphs:
1. A combination of CAQDAS (Atlas.ti) and a manual process will be used to code the data.
2. Since the financial accounting examinations follow a very similar and traditional layout (see
Chapter 4), the assessment tasks, which is the “required” part of the examination (which
follows the “information given” or “case study” part) will be carefully analysed in terms of
the SOLO taxonomy, by amongst other things considering the verbs used in these
assessment tasks. The analysis will revolve around what the assessment task (the
“required” part) really demands in light of the information given and the SOLO taxonomy.
The assessment tasks will then be labelled under a SOLO level (a priori code) together
with the type of assessment task (which will represent a code derived from the data). This
process will lead to themes that can be reported and also used to solve (or at least
decrease the scope) of the problem of assessment tasks that use verbs which are not
easily categorised (or too arbitrarily categorised) under the revised Bloom’s taxonomy.
3. During the analysis, concepts that seem to pertain to the same phenomenon will be
grouped under a code that ultimately feeds into a theme. De Vos et al. (2018:411) refer to
this process as “discovering categories”. This is an example of codes that will be derived
from the data by the researcher (see Figure 3-3, page 75). These codes will then further
be analysed in light of their effect (or lack of effect) on the SOLO level of the examinations.
4. The literature around SOLO (Chapter 2) will also be used to deduce codes for testing
against the data, as well as to induce ideas (observations), and to generate codes or
themes which may be applicable to the data.
3.4.4.5 Thematic analysis
As mentioned in Table 3-6 (page 68), thematic analysis entails the search for and recognition
of patterns within the data. These patterns form themes that become the categories for
analysis (Bowen 2009:32). A theme (i.e. the patterns) is a broad category of several related
codes (or even just one prominent code) indicating an idea which is important to your research
question (Saunders et al., 2016:584).
Saunders et al. (2016:579) emphasise the importance of thematic analysis by seeing it as a
“foundational method for qualitative analysis”. Saunders et al. (2016:587) also motivate the
77
use of thematic analysis as an accessible and flexible technique, which, compared to some
other qualitative approaches, is not overly prescriptive about the application of its analytical
procedures. Thematic analysis is found to be an adaptable approach suited to deductive as
well as inductive approaches (Saunders et al., 2016:587).
The recommended procedures (Saunders et al., 2016:580) for thematic analysis are (with own
comments given in brackets):
becoming familiar with your data (this process happens throughout and is crucial);
coding your data (coding forms a substantial part of thematic analysis and is explained
above);
searching for themes and recognising relationships (a good way to do this is to keep
asking questions, e.g. What are the key concepts in these codes?; What, if anything,
seems to be recurring in these codes; Which codes appear to be related?; What
patterns or trends are evident in the coded data? (Saunders et al., 2016:584-585));
refining themes and testing propositions (this process typically involves the combining,
splitting or discarding of themes or re-analysing of the data (Saunders et al., 2016:585),
however, the aim is still to answer the research questions credibly).
These steps will be followed in this study.
3.4.4.6 Content analysis
The process of content analysis, and content analysis as applied in terms of this study, is
thoroughly described in paragraph 3.4.1 (under the discussion of mixed methods research)
and paragraph 3.4.2 (as being the quantitative design type for this mixed methods study). The
data analysis technique was set as one of basic descriptive statistics (the central tendency
and the dispersion) on the quantitative data derived from the content analysis process, which
can include the amount of time per assessment task or the number of pages of reading
material or other frequencies that will be discovered. The aim is to utilise this limited but
potentially influential quantitative data to support the qualitative themes emanating.
3.4.4.7 Textual analysis
In Table 3-7 (page 69) it is stated that a part of the analysis process of this study will be in the
form of textual analysis. Textual analysis involves the analysing of text (religious or literary) in
order to understand the meaning of such texts (Mouton, 2013:167). The emphasis is less on
the number and frequency of occurrences but more on interpreting the meaning a document
may have (De Vos et al., 2018:381). This process also involves “reading between the lines”.
78
This researcher will draw on his financial accounting and assessment experience in order to
further obtain meaning from the text in the financial accounting examinations. The main aim
here is to assist the coding process to determine what the assessment tasks (the “required”
part) really demand in light of the information given as well as the SOLO taxonomy.
A court case that this researcher encountered during a professional engagement provides a
good definition of how documents can be interpreted. This “golden rule” is found in par. 18 of
the judgement in the case Natal Joint Municipal Pension Fund v Endumeni Municipality (2012).
The summary of Wallis (2019:22) is given, with own emphasis added in italics:
Interpretation is the process of attributing meaning to the words used in a document, … having
regard to the context provided by reading the particular provision or provisions in the light of
the document as a whole and the circumstances attendant upon its coming into existence.
Whatever the nature of the document, consideration must be given to the language used in the
light of the ordinary rules of grammar and syntax; the context in which the provision appears;
the apparent purpose to which it is directed and the material known to those responsible for its
production. … The 'inevitable point of departure is the language … itself', read in context and
having regard to [its] purpose … and the background to the preparation and production of [this]
document.
The textual analysis will be done bearing the above in mind.
3.4.4.8 Reliability and validity
All researchers should take issues of research quality seriously if they wish others to accept
their research as credible (Saunders et al., 2016:204). Traditionally, the quality of research
was assessed as a function of having “reliability” and “validity”, which as concepts initially refer
to the positivist tradition as found in the natural sciences while using a quantitative approach
(Saunders et al., 2016:202). Reliability refers to a factual attribute of whether the executed
research methodology will consistently provide the same findings (replicability) for a given data
set, while validity refers to appropriateness of the measures used, accuracy of the analysis of
the results and generalisability of the findings (Saunders et al., 2016:202).
Scholars using an interpretive approach under qualitative research have however adapted
and expanded these concepts to rather be utilised under the premises of dependability,
credibility and transferability (Saunders et al., 2016:202). The reasons for this include that an
interpretivist philosophy provides for multiple realities or meanings, subjectivism and inductive
reasoning, to name a few (refer to Table 3-2, page 52), all of which are anti-positivist.
79
Reliability is therefore paralleled to “dependability”, which refers to the researcher’s
responsibility to describe the ever-changing research context that is typically found during
qualitative interpretivist research (Trochim & Donnelly, 2008:149). Since this study does not
have direct participants (who may have changing views), and since a great amount of context
is described throughout, dependability is not seen as a concern.
“Credibility” is seen as the parallel of internal validity (Trochim & Donnelly, 2008:149) and
involves establishing that the results of the qualitative research are believable or credible from
the perspectives of the participants, since their socially constructed realities are represented
by the research (Saunders et al., 2016:206). Since this study does not have direct participants
whose views are represented, it is submitted that it is prudent for the focus to rather be shifted
to what may be called “overall credibility”, which is accepted to mean whether this research
will be believable (within the philosophy and parameters given) by a reasonable and informed
reader. This “overall credibility” is deemed to include the issue of transferability (the parallel of
external validity), which in essence refers to demonstrating transparency of process, typically
by means of thorough description (Saunders et al., 2016:206). Saunders et al. (2016:206)
describe “transferability” as providing a full description of the research questions, design,
context, findings and interpretations, so that the reader has the opportunity to judge the
transferability of the study to another setting in which the reader is interested to research. This
process does not reach “generalisability”, as is expected with statistical quantitative research,
but refers to a process of disclosure so that the research can be judged as credible and
therefore usable or not.
For this study the view was taken that credibility and validity (meaning transferability and
usability) should be considered throughout every phase of the study. The aim is that the study
should also be of practical use. Reliance is mostly placed on the rigorous description of the
research design, context and methods as recommended by Saunders et al. (2016:206).
Further examples of where credibility and validity are discussed can be found in every phase
of this study. These examples include:
Consideration of other scholars’ work in Chapter 1 and Chapter 2.
Careful consideration and framing of the study under accepted philosophical
assumptions (Chapters 1 and 3).
In Table 3-7 (page 69) it is motivated that the data of this study is non-reactive.
The study will be reviewed by an experienced supervisor.
From a viewpoint of credibility, it is also acknowledged that the fact this researcher is solely
responsible for coding the data may be perceived as too subjective, even if the interpretive
80
paradigm is invoked as the stated philosophy. A further defence, as was found in Van Rooyen
(2016:319), is therefore given in that the researcher having ten years of financial accounting
academic experience has drawn on this relative expert experience to code as objectively and
technically sound as possible. Furthermore, there is no incentive for the researcher to
manipulate the coding process. The data-specific details of the coding process and its
corresponding findings is also richly described (for purposes of transparency) in Chapter 4.
3.5 SUMMARY
For any research project to be credible, the research design and methodology has to be
robust. The research design and methodology for this study were therefore systematically
described and defended in this chapter. The chosen design and methodology were driven by
the research objectives as underpinned by an overall philosophy of pragmatism. The nature
of the research objectives and data lead to a methodological choice of mixed methods
research, with the qualitative research design being dominant. The qualitative design type was
motivated as document analysis specifically set in an interpretative paradigm, mainly relying
on the technique of thematic analysis. The crucial process of coding of the data was also
addressed. The quantitative part of the study was motivated to utilise content analysis and
basic descriptive statistics as part of the overall pragmatic approach to achieving the research
objectives. The chapter concluded with a discussion of the concepts of reliability and validity,
as pertaining to the study. In the chapter that follows the research design and methodology
will be effected so as to generate the empirical evidence that is needed to meet the research
objectives.
81
CHAPTER 4 ANALYSIS AND FINDINGS
4.1 INTRODUCTION
In Chapter 3 above, the chosen design and methodology (including the paradigmatic
assumptions) of this study were thoroughly motivated and defended.
The purpose of this chapter is to document the empirical evidence generated in order to
achieve the following research objectives:
As part of the proposed framework, typical Financial Accounting 1 summative
assessment tasks with the analysis through the SOLO taxonomy thereof will be
documented (research objective 3).
Typical themes that have an influence on the standard or level of a Financial
Accounting 1 summative exam will be determined by analysis through the SOLO
taxonomy (research objective 4).
The insight gained through the study will be synthesised to flow into the framework
that will be given in Chapter 5 and which can guide the application of the SOLO
taxonomy in evaluating the standard of summative assessment of a Financial
Accounting 1 module (research objective 5).
In starting the data analysis process, the content analysis technique (as described in
paragraphs 3.4.1 and 3.4.2 above) was chosen, since it will aid the researcher in getting more
familiar with the data as well as immediately demonstrate any strange occurrences (e.g. words
that one won’t necessarily expect in a Financial Accounting 1 examination).
4.2 CONTENT ANALYSIS APPLIED IN SUPPORT OF THE THEMATIC ANALYSIS
In paragraphs 3.4.1 and 3.4.2 above it is described that content analysis will be performed as
part of the bigger data analysis process. The content analysis was also described as a process
of “quantifying” some qualitative data. In terms of this study it entails that, at a basic level, the
number of times a word occurred (frequency) is analysed. Figures 4-1 and 4-2 (pages 86-87)
and Tables 4-1 and 4-2 (pages 86-87) are the first products of this content analysis with the
results discussed below. The second part of the content analysis, namely the “number of
pages of reading material”, is incorporated in Table 4-3 (page 90) which comprises the
quantitative part of this study as was analysed through the basic descriptive statistics that was
motivated in paragraph 3.4.2. It is important to remember that both the content analysis as
well as the quantitative part of the study has a supporting role and are not the primary
82
techniques of analysis. The notion of a word count may seem over-simplistic on the surface,
but in terms of this study it has an important function in terms of enhancing credibility (of
observations) and in terms of supporting the themes that follow.
It will now firstly be explained how Figures 4-1 and 4-2 (pages 85-86) and Tables 4-1 and 4-2
(pages 85-86) were created, after which an analysis will be given.
Figure 4-1 (page 85) was created by importing the seven universities’ Financial Accounting 1
examinations into the Atlas.ti CAQDAS software. The software was then used to count the
number of times a certain word was used. This word count or word frequency is then
represented visually in a “Word cloud”, with the more frequent words having a larger font and
being placed in the middle of the figure. For purposes of transparency and clarity, Table 4-1
is given as an extract of the statistics on which Figure 4-1 is based on. Table 4-1 shows the
frequency of each word from highest to lowest, as well as the particular word’s frequency as
a percentage of the total number of unique words. Single character words, numbers, hyphens
and underscores were not counted (as was suggested by the software), since there is no
meaning to be found in such items. The software does not regard capital letters of words such
as “May” or “December”, and instead converts all words to the lower-case form unless an
acronym is used, in which case it will be reported as an all-capital letter word. The software
also allows the user to trace a word to its specific exams. Figure 4-1 visually only shows words
that occurred 31 times or more, which is just above four times per exam, and was driven by
the software suggesting the optimal visuals. Figure 4-1 does not change the underlying
frequency of the words.
Figure 4-2 was created following the same procedure and settings as for Figure 4-1, but with
the difference that only the assessment tasks as represented by the “Required” part (see
Figure 4-3 below) were imported into Atlas.ti. Due to the lesser number of words as compared
to the full exams, Figure 4-2 shows all words occurring at least seven times (as suggested by
the software) which is on average once per exam. Figures 4-1 and 4-2 will therefore also have
some words that overlap, e.g. the word “prepare”.
The analysis of Figure 4-1 and Table 4-1 is done in the context of this researcher’s experience
as an accountant and lecturer. In analysing Figure 4-1 and Table 4-1, it can be seen that the
word “year” is most frequent with months’ names, e.g. “may” or “december” also being
materially used. This is consistent with financial accounting being a period-driven discipline,
e.g. yearly reporting. Figure 4-1 shows that the exams keep to the topics around financial
accounting and does not contain any unexpected occurrences such as swearing. The words
83
“question” and “required” refers to the layout of the exams (see Figure 4-3, page 91) while
Figure 4-1 (page 85) also shows for example that spelling mistakes are not prevalent.
Furthermore, one can also see that financial accounting, as expressed through these exams,
is concerned with:
business types (e.g. the words “ltd” or “partnerships”)
financial statements (e.g. the words “statement” or “financial” “position”)
supporting records (e.g. “general” “ledger” or “journal”)
the typical topics or elements found in a financial accounting system. These include
“bank” and “cash”; “assets” such as “debtors”, “furniture”, “equipment” and “inventory”;
“liabilities”; “shares” and “capital”; “income”, “expenses” and profit” etc.
In this researcher’s experience the manifestation of these items are not a surprise and is part
and parcel of a typical Financial Accounting 1 curriculum. These items also form part of the
fundamental concepts of accounting (see discussion below) and are also found throughout
IFRS (see IASB, 2018 as an example).
Moving on to Figure 4-2 (page 86) and Table 4-2 (page 86), the standout word is “prepare”.
This is the verb that was materially used (in frequency) to state the assessment tasks, and is
not a surprise since it incorporates most of the tasks of a real-life financial account, e.g.
preparing the journal entries or financial statements. What Figure 4-2 thus supports is the
research problem as discussed in paragraph 1.2, in that it is not easy to use the verb “prepare”
to classify a financial accounting assessment task on the revised Bloom’s taxonomy. This verb
can imply many different levels of complexity, and therefore the SOLO taxonomy is of interest.
Other verbs that were used, albeit much less frequently, include “calculate” (19 times),
“discuss” (10 times) and “disclose” (once). These verbs have the same implication in terms of
the revised Bloom’s taxonomy as “prepare”.
The remaining words that are found in Figure 4-2 share the analysis that was given for Figure
4-1, but it is apt to mention again that the words “question”, “required”, “marks” and “minutes”
refer to the layout of the exams (see Figure 4-3, page 91) as well as issues of time and grading
(see paragraph 4.3.1).
Based on i) Figures 4-1 and 4-2, ii) the review of the exams, and iii) the researcher’s
experience, it can be seen that the financial accounting exams and the assessment tasks all
tie up to what is defined here as the “fundamental concepts of accounting”. IFRSs (IASB,
2018) are also directly structured around these fundamental concepts. These concepts will
84
assist in evaluating the SOLO level of the assessment task (as shown in the paragraphs that
follow). These concepts are also typically found in most definitions of “financial accounting”.
An example of such a definition can be found in a prescribed textbook (Myburgh et al., 2012:2)
for the NWU’s Financial Accounting 1 students, which describes financial accounting as
follows:
“The practice of and knowledge of systematically identifying, measuring,
recording and reporting of quantitative information, which is primarily financial in
nature, concerning economic activities of entities.”
The aforementioned process and other sources (Cloete & Marimuthu, 2015:1; IASB, 2018)
brought the following main concepts to the fore:
Financial Accounting being:
1. a process or system of
2. identifying and gathering,
3. recognising and recording,
4. measuring,
5. summarising and classifying, and
6. presentation and disclosure of business transactions and financial information that
ultimately feeds into financial statements that states the business’s performance and
financial position
7. and enables interpretation of the above.
This statement of fundamental concepts concludes (except for Table 4-3, page 90) the applied
content analysis part of this study. In the paragraphs that follow, the thematic analysis is
applied as part of the bigger process of analysing the documents (i.e. the financial accounting
examinations).
85
Figure 4-1: The “Word Cloud” illustrating the word frequency in the complete examinations
Source: Researcher
Table 4-1: An illustration of the word frequency in the examinations
Word Total %
year 120 2.56%
statement 96 2.05%
financial 91 1.94%
may 86 1.83%
account 84 1.79%
cost 81 1.73%
following 81 1.73%
ltd 80 1.71%
balance 79 1.69%
bank 79 1.69%
cash 79 1.69%
question 74 1.58%
profit 70 1.49%
information 69 1.47%
income 68 1.45%
shares 66 1.41%
Source: Researcher
86
Figure 4-2: The “Word Cloud” illustrating the word frequency in the examinations’ assessment
tasks
Source: Researcher
Table 4-2: An illustration of the word frequency of the examinations’ assessment tasks
Word Total %
prepare 56 2.00%
statement 49 1.75%
marks 47 1.68%
required 42 1.50%
question 40 1.43%
year 38 1.36%
account 36 1.29%
ended 35 1.25%
cash 32 1.15%
general 30 1.07%
income 29 1.04%
asset 28 1.00%
profit 28 1.00%
information 27 0.97%
financial 24 0.86%
May 22 0.79%
Source: Researcher
87
4.3 ANALYSIS OF ASSESSMENT TASKS AND THE EMANATING THEMES
The following paragraphs explains the results of the thematic and textual analysis.
4.3.1 Theme 1: Time allowed for the assessment tasks
The first theme (“Theme 1”) that is immediately obvious as a repeating pattern across all the
examinations is the “time allowed for the assessment tasks”, as shown in Table 4-3 (page 90).
It is noted in paragraph 2.5.2.1 above that time stress tends to make students revert to a lower
level of cognitive response, which by analogy means that the students may operate at a lower
SOLO level than what was intended or expected by the assessment task. A recent observation
by this researcher seems to support this notion in that students who submitted their tests under
no time limit during the first semester of 2020 (under the COVID-19 lockdown) wrote much
higher quality (completeness and substance) answers compared to their first assessment in
the second semester, where the normal time limits where applied. Of course, many variables
can and do have an influence, but since the students themselves admitted the huge impact of
time stress, it is valuable to take their experience into consideration, especially under the
interpretive paradigm. Therefore, a further analysis of issues around time now follows.
In analysing Table 4-3, it can be seen that there are differences in the time allocated to the
assessment tasks (questions) by the different universities. This is not necessarily wrong or
unfair, since there are many other factors influencing the SOLO level; for example, time
management in itself may have been an intended learning outcome influencing one exam,
while it may not have been an intended learning outcome for another exam at that moment in
the qualification.
It can be seen that Universities D, F and G on an overall level allow the least amount of time
per mark. It is submitted that Universities D and F compensated for this by having shorter
questions which is inherently easier to digest, while University G has less reading material.
The mark allocation during grading of answers may also be a consideration and while in
general one mark is awarded per answer, Universities B and F award half marks for some
answers (e.g. per accounting amount), while University D awards two marks for some answers
(i.e. the multiple-choice questions). In this researcher’s experience, students are informed of
the grading practices at each university and hence are not at a disadvantage because of this.
A student is also expected to prepare the required answer as indicated by the exam
irrespective of the exact finer detail on the grading plan of how the marks are distributed.
88
Furthermore, it can be seen that University E tends to include longer questions, i.e. those that
count more marks per question. University E has one question counting 37 marks. Longer
questions or more pages of reading material generally have an effect on time pressure, since
students normally have to spend more time to plan their responses (answers) or spend more
time reading. One should, however, be careful to express a view based on this finding that
University E’s exams are more difficult or better or unfair, since there are many factors to
consider (such as the nature of the question or intended learning outcomes). For the purposes
of this study, the issue here is the time pressure effect on the SOLO level and not whether the
time allocation is correct or not.
The numbers in Table 4-3 (page 90) do however show that all the universities set their
examinations within a reasonable range of time allocated, which is expected, since the
examinations are all stated as testing on an NQF 5 level and covering the same financial
accounting concepts.
Furthermore, it is noted that an argument can be built that time pressure may help some
students to focus better on the task at hand, which may then compensate for the effect of the
time-stress response. This increased focus is unlikely to change the SOLO level required by
the assessment tasks and is not likely to be a default response in contrast to time-stress. The
aforementioned is supported by the findings of Lucas and Mladenovic’s (2009:263) study.
Further studying of such a phenomenon is however beyond the scope of this study.
In conclusion of this theme it is noted that Biggs and Tang (2007:199) suggest that in order to
derive a maximised benefit from applying constructive alignment in the teaching and learning
process, assessment should only be time constrained when the target performance itself is
time constrained. Examples of such a target performance include a lawyer’s performance in a
court room or a surgeon performing surgery. Biggs and Tang (2007:199) suggest that
assessing such performance is best done by going beyond written examinations. Financial
accounting as a professional discipline probably leans towards having a target performance
that is subject to time constraint since “time is money”, but fortunately (since written
examinations are so prevalent), and due to its inherent nature (of working with numbers,
financial reporting, using supporting documents, etc.), written examinations can simulate real
life performance to a great extent.
The summarised findings of Theme 1 (that is carried forward to Chapter 5) is thus that
examiners must carefully contemplate whether an assessment task is to test time
management skills, reading skills, planning skills, the concepts (and topics) of financial
accounting, or all of these, as well as to what degree. Examiners must be careful that time
89
pressure does not inadvertently frustrate the SOLO level at which the assessment task aims
the students to operate at.
90
Table 4-3: Analysis of the time allowed for assessment tasks of the sample of financial accounting exams
UNIVERSITY A
UNIVERSITY B
UNIVERSITY C
UNIVERSITY D
UNIVERSITY E
UNIVERSITY F
UNIVERSITY G
Total marks 100 100 80 150 160 100 125
Total minutes 180 180 120 180 240 120 150
Minutes per mark 1.8 1.8 1.5 1.2 1.5 1.2 1.2
Total number of pages of reading material
9 6 5 13 15 7 6
Time indicated (or can easily be derived) for sub questions
Yes Yes Yes Yes Yes Yes Yes
Number of questions and sub-questions*
21 11 10 46 12 14 12
Average marks per sub question*
4.8 9.1 8.0 3.3 13.3 7.1 10.4
Longest question in terms of marks
17 21 20 16 37 21 27
Shortest question in terms of marks
1 2 2 1 2 1 3
*University A's paper contained 8 x 1-mark multiple-choice questions. University D's paper contained 15 x 2-mark multiple-choice questions. University F's paper contained 5 x 1-mark multiple-choice questions. Source: Researcher
91
Question 1 (x marks, x minutes)
Descriptive information / Case study
Descriptive information / Case study
Descriptive information / Case study
Required:
Assessment task
Assessment task
4.3.2 Theme 2: The layout of the assessment
Relating to Theme 1 (as explained below), but having distinct implications, is the second theme
that was identified, being a repeating pattern across all the examinations, as “the layout of the
assessment”.
Ambrose and Harris (2005) define “layout” as “an arrangement of parts etc. according to a plan”.
Ambrose and Harris (2005:5) also describe layout as the placement of text and image elements
within a design. They assert that how these elements are positioned will affect how the content is
viewed and received by the readers, as well as their emotional reaction towards it (researcher’s
own emphasis). They note that layout can help or hinder the receipt of the information presented
in a work, and state that creative layouts can add value and embellishment to a piece, whereas
understated layout can allow the content to shine through (2005:5).
The examinations for Universities A to G are highly similar in terms of layout. Except for relatively
minute detail such as font size and font formatting, the layout of all seven universities’ exams are
in essence the same. This typical layout is shown in Figure 4.3 below. Although the exact reasons
for this similarity is beyond the scope of this study, Figure 4.3 shows that Ambrose and Harris’s
(2005:5) statement that “understated layout can allow content to shine through” may well be
applicable. One should also note that these examinations are part of the process for students to
eventually write the final professional examinations, and hence in this researcher’s experience
these examinations seek to simulate the layout of the professional examinations. The explanation
given below of the possible effect that layout has on the SOLO level of an assessment task
logically explains why a first-year exam may simulate professional examinations in terms of
layout.
Figure 4-3: The layout of Universities A to G’s examinations
Source: Researcher
92
In paragraph 2.5.1.2 above it can be seen that cognitive demands increase with the SOLO level
of an assessment task. In paragraph 2.5.1.3 and Figure 2-3 (page 44) it is also shown that there
is a distinct change in the progression from the multi-structural level to the relational level. The
relational level heralds the qualitative phase of cognitive demand where detail can be integrated
into structural patterns and understanding is increased as opposed to a mere quantitative
increase in knowledge. Therefore, looking specifically at the multi-structural and the relational
levels, it was found (see the illustrative scenario below) that a layout which students perceive as
unfamiliar or complex, can have the effect of raising the SOLO level of the assessment task for
that student. This is because the effect of the layout of the examination in itself then becomes a
concept to cognitively process along with the other requirements of the assessment task. A stated
assessment task can only be understood if the given information, as laid out, is successfully
cognitively processed by the student.
To illustrate this effect of layout, the following scenarios serve as an example, which was
constructed based on University B’s question 1 (this university and question was chosen at
random) as well as some real-life experience of this researcher.
At this point it should be noted that going forward, any information pertaining to the year of an
exam question (e.g. 2016) was substituted by “20XX” so that the ethical principle (as discussed
in paragraph 1.5) of de-linking the universities from the data analysis and findings remains
adhered to. For the same reason any organisational names will be replaced by for example “A
(Pty) Ltd” or “B General Dealers”.
Scenario 1:
The question from University B provides a trial balance and list 13 points of additional information
(i.e. potential adjustments needed) about a partnership.
The assessment task requires: “By taking into account the above adjustments, prepare the
statement of profit or loss – including the distribution section – for the year ending on 28 February
20XX in order to comply with IFRS. Ignore VAT”. The question counts 21 marks.
The verb “prepare” does not clearly place this assessment task into a SOLO level (which is partly
how this study’s problem statement originated, as given in paragraph 1.2 above).
Ignoring (for now) other possible factors as outlined in this study, this assessment task is analysed
as follows:
There are 11 marks that are available for restructuring the trial balance into a statement
of profit or loss. The main financial accounting concept (see paragraph 4.2 above) that the
93
assessment task test is the concept of “presentation and disclosure of financial
statements”.
There are 10 marks available for the adjustment of the values of the given accounts by
processing the 13 points of listed information given. These adjustments mainly test the
financial accounting concept of “recognising and recording of financial information”.
Although the way this scenario is analysed will be further refined as the study progresses, one
can state that 11 marks of the question is mostly on the multi-structural level, since at least two
concepts are tested, namely “presentation and disclosure” of the statement of profit or loss but
also the “classification” of the accounts into the correct line item of the statement of profit or loss.
It must be remembered that, as pointed out in paragraph 2.5.1.3 above, Biggs and Tang (2007:78)
note that today’s SOLO level may be tomorrow’s lower SOLO level as new knowledge becomes
conventional wisdom (this theme is explored as Theme 3 below). The classification of accounts
into the line items of a financial statement is an example of conventional wisdom in terms of
Financial Acounting 1. In line with the description of the SOLO levels in paragraph 2.5.1.3, it can
be seen that this assessment task also does not require a deep understanding or qualitative
increase of knowledge, and therefore cannot easily be justified as being on the relational level.
The 10 marks available for the adjustment of the values of the given accounts by processing the
13 points of listed information given, can thus be confidently stated as being on at least the multi-
structural level, since at least two accounting concepts are tested. Some of the marks may
however qualify as being on the relational level, since the adjustment of the accounts involves a
relational aspect in terms of how the adjustment relates to certain accounts or line items. This will
however depend on the “conventional wisdom” (see Theme 3 below) or “familiarity” of the student
with the given adjustment. For the purposes of this example is not necessary to elaborate further.
Scenario 2:
The same assessment tasks as given in Scenario 1 applies. The only change is that the trial
balance and adjustments are now not given in a straight forward layout. The information is given
across multiple pages in the format of e-mails, “WhatsApp”-messages and a dialogue between
two parties.
It is submitted that the majority of the marks now clearly moves into the relational level, since the
layout of the assessment task in itself is a concept that must now be processed by the student,
seeing as the student must relate (categorise) the information correctly before the can even start
classifying and disclosing. This assessment task however does not move into the extended
abstract level, since students should still be able to recognise the concepts of the trail balance
94
and adjustments, even if under a different layout, as “seen-before” information – they are not
required to function in a totally new and unknown domain (the so-called “fourth dimension”).
This illustrative scenario explains this researcher’s experience that students often comment as
follows after they have written an examination: “Sir the exam was OK but question 4 was asked
in a funny way …”. When the suggested solution is shown to the student, the reaction is typically
one of “Wow, it was so easy, I can’t believe I stressed so much while completing the question”.
As discussed above, the layout of an assessment also influences how a reader reacts to the
content of the assessment, and thus may also have an effect on the time allowed to complete the
assessment, since a complex or unfamiliar layout will require more time for the student to function
on the relational level (i.e. more “thinking time” is required).
Therefore, the summarised findings of Theme 2 which are carried forward to Chapter 5, is that a
layout that students perceive as unfamiliar or complex can have the effect of raising the SOLO
level (which may be desired or not) of the assessment task for that student, because the effect of
the layout of the examination in itself then becomes a concept to cognitively process along with
the other requirements of the assessment task. A complex or unfamiliar layout will also affect the
time allowed to complete an assessment, since such a layout will require more time from the
student in order to function on the relational level (i.e. more “thinking time” is required).
4.3.3 Theme 3: Conventional wisdom
Before discussing this theme, it is prudent to explain that, since it borders on crossing deeply into
the specialised field of cognitive psychology, which is beyond the scope of this study, this
researcher’s own experience, observations and reflections are drawn upon in exploring it.
Readers of this study should still bear in mind that this study is mainly written under the interpretive
paradigm, wherein this researcher accepts that other realities may exist.
The genesis of this theme then, is Biggs and Tang’s (2007:78) pertinent statement that “today’s
extended abstract is tomorrow’s relational”. Biggs and Tang (2007:78) give two examples in
stating: “Marton and Säljö’s original study was such a breakthrough; linking approaches to
learning to systems theory was another, but now both are conventional wisdom”. This theme
therefore also connects to the teaching strategy of building on prior knowledge (“working from
known to unknown” or “scaffolding”), wherein it is widely accepted that building on quality prior
knowledge positively influences i) further knowledge acquisition and ii) the capacity to apply
higher-order cognitive problem-solving skills (De Corte, 1990:237).
95
In the South African higher education system, when an assessment task is evaluated in order to
categorise it under a Bloom or SOLO level, it is done in the context of an NQF level. What is
implied is that a student has had a reasonable amount of effective facilitation, guidance,
preparation, access to the necessary resources and other interventions so as to be able to handle
a certain NQF level’s requirements. Therefore, examiners, in an environment of constructive
alignment and fair assessment, may for example justly evaluate that a particular assessment task
is on a multi-structural or relational SOLO level, but the risk is that these conditions (i.e. effective
facilitation, guidance, preparation, etc.) may be absent for a certain student, which then has the
effect that the SOLO level of that assessment task becomes one of extended abstractionism for
that student. This may typically lead to a mismatch between the intended and perceived standard
of the assessment (for that student), which can then lead to a range of unwanted complications
such as low grades. Such a scenario also typically has implications for time pressure and mental
stress, which may further exacerbate this effect. It is furthermore submitted that this risk is a
primary reason why students of financial accounting is constantly encouraged and assisted to
study the module content and practice answering problem-based questions. It is also one of the
main reasons that continuous assessment is applied before the summative assessment.
This theme will now be demonstrated with a scenario from one of the financial accounting
examinations to support the above explanation (the university and question was randomly
chosen):
Scenario and discussion:
The question form University G provides half a page of information (i.e. the “case study”) about a
company that manufactures plastic bottles and sells them to various water retailers.
The assessment task requires: “Using the five-step model of IFRS 15, discuss with reasons the
recognition of revenue from the contract with G (Pty) Ltd. Assume that the agreement between G
(Pty) Ltd and Z (Pty) Ltd satisfies the requirements of step 1 of IFRS 15.” The question counts 6
marks.
This researcher’s experience dictates (and most financial accounting lecturers should concur)
that this is a classic Financial Accounting 1 assessment task, which entails the listing of rules and
the relation or application of given information to those rules while ultimately concluding with an
answer. Even though the word “relating” is used, this does not place the assessment task on the
relational SOLO level. If one analyses the assessment task by also taking into account the
concepts of financial accounting (see paragraph 4.2 above) the task actually entails two broad
concepts, which are i) “recognising and recording” of ii) one standalone topic which is either sales
96
or revenue. The task is thus seen as multi-structural if a student has for example seen it before,
has practiced it, or has previously been guided on a similar task. In this instance there will be little
relational aspect (which does not mean it is a “bad” or invalid task). The reason for this is that
typically a student would list the five rules (in this case of IFRS 15) and then scan for the typical
“sales contract” information and list that under each item of the first list. Had the assessment task
brought in an element of another topic, for example taxation or inventory, then the task will
probably be on the relational SOLO level, since then an “autopilot” response by the student will
be unlikely to actually be required by the task. It is also interesting to see above that the first step
of the assessment task is given, for which there may be a few reasons, but ultimately it may be
understandable since it is an introductory (in terms of becoming a professional) financial
accounting examination after all.
However, if for example the nature of the topic (i.e. IFRS 15 Revenue from Contracts with
Customers) is new (as was indeed the case when this exam took place) and students, facilitators
or even examiners have not had much chance to develop relative conventional wisdom over the
topic, then the assessment task becomes clearly relational in terms of SOLO. This is because the
student will have to process each part of the given information carefully and relate it to the correct
step in the list for revenue recognition. If a student for some reason missed the IFRS 15 lecture
and had no other effective facilitation or study guidance, then the whole assessment task may
become extended abstractionism for that student which, combined with time pressure and mental
stress, will likely lead to failure.
Admittedly this theme involves a degree of professional judgment from the examiner, since the
examiner, while regarding the best available information, will have to make a call on the extent
and effect of “conventional wisdom” on the particular assessment task. This element of judgement
should not concern the reader of this study, since logically evaluating any assessment is
inherently interpretative by nature as one deals with human beings and their personal and
subjective experience and behaviour. The concepts of “cognitive demand” or “level” or “standard”
of assessment tasks are all quite qualitative, subjective and intangible in nature and not
susceptible to exact quantification (Biggs and Tang (2007:171) make a similar argument). Hence
the aim is one of more objectively measuring the level (or standard) of a summative assessment
and not claiming to find a fully objective method. This stance is underpinned by the interpretative
paradigm.
In conclusion of this theme it is therefore suggested that the examiner should, while regarding the
best available information, consider how much conventional wisdom a well-prepared student (i.e.
the “reasonable man” test) should possess in terms of a certain assessment task. The examiner
97
should then consider the effect of the conventional wisdom on the SOLO level of an assessment
task.
The above findings of this theme will be taken forward and summarised in the framework in
Chapter 5.
4.3.4 Reflecting on Themes 1 to 3
In paragraph 3.4.4.4 it is explained that qualitative data analysis is mostly a process of analysing
and re-analysing data (instead of being a linear process) since while one codes, new
understandings emerge from the data, which may alter the codes themselves or the relationships
between the different pieces of data. Therefore, in terms of this study, the need was felt to further
reflect on Themes 1 to 3 so as to document even more meaning and understanding.
It is apt to firstly briefly summarise how Themes 1 to 3 originated. Theme 1 originated as a set of
recurring patterns (coded as “Time allowed”) that were observed but are also grounded in
literature as seen in paragraph 4.3.1. Similarly, Theme 2 originated as a set of recurring patterns
(coded as “Layout”) that were observed but are also grounded in literature as seen in paragraph
4.3.2. Theme 3 (“conventional wisdom”) was however induced by the literature and the
researcher’s experience as is explained in paragraph 4.3.3 above.
The common thread that can be seen in the explanation of these three themes is that they are
more likely to change (or “disrupt”) a SOLO level of an assessment task than to set it. Of course,
they can sometimes help to set a SOLO level, for example where the time pressure or layout is
an intended learning outcome to be assessed in itself or where students are purposely required
to function on an extended abstract level through the design of an assessment that is clearly
beyond “conventional wisdom”. However, in general the implication is that in most cases a SOLO
level will be set (as explained in Theme 4 below) after which the impact of Themes 1 to 3 should
be considered. Themes 1 to 3 can therefore be further categorised as the “disruptor themes”.
During the analysis of Themes 1 to 3 (paragraphs 4.3.1 to 4.3.3) the fundamental concepts of
accounting (paragraph 4.2) as representing cognitive demand could be used sufficiently to show
the impact of these themes on the SOLO level of an assessment task, and even illustrate the
instance where these “disruptor themes” could possibly outright set the SOLO level of an
assessment task. The issue found with the analysis applied under Themes 1 to 3 is that it was
not yet a structured process that is geared to set an initial SOLO level for an assessment task.
Therefore, drawing heavily on experience, the review of the data and asking the basic question
of “how can this be better structured?”, as well as considering all the information given in this
98
study up to this point, Theme 4 was formalised, which is termed the “dimensional method of
applying the SOLO taxonomy to the assessment tasks of Financial Accounting 1”.
4.3.5 Theme 4: The “dimensional method” of applying SOLO to the Financial
Accounting 1 examinations’ assessment tasks
What the researcher realised in utilising the SOLO taxonomy is that the notion of “the cognitive
demand induced by the assessment tasks” is central. This notion can be found throughout this
study with the prime example being evident from paragraph 2.5.1.3 and Biggs and Tang’s (2007)
work being the prime source. One can for example see this in Biggs and Tang (2007:11) where
they state that (own emphasis added):
“Good teaching is getting most students to use the level of cognitive processes needed to
achieve the intended outcomes that the more academic students use spontaneously.”
The SOLO taxonomy (and its levels) is thus a proxy of cognitive demand with the suggested way
to set the SOLO level to find the verb (e.g. “apply”) of an intended learning outcome or student
response. However, with the verbs not being a practical option in terms of Financial Accounting
1 assessments (as discussed in paragraphs 1.2 and 4.2 above), another credible way to set the
SOLO level for the financial accounting assessment tasks had to be found.
After specifically and repeatedly analysing Figure 2-3 (page 44) and giving this study many hours
of thought, it was realised that the following “elements” could provide a “dimension” which can
link the assessment task efficiently to the SOLO taxonomy.
The elements to consider were found by asking “what is it that this assessment task requires the
students to process cognitively?”
The main elements were found to be:
The fundamental concepts of accounting (see paragraph 4.2) that an assessment task
requires (e.g. recording or classification or reporting, etc.).
The topics of financial accounting that are involved, e.g. bank and cash or inventory, etc.
These topics would for example be derived from the ILOs, which constitute the curriculum.
Some of these topics are noted in paragraph 4.2 and shown in Figure 4-1 (page 85).
Examiners should once again note that this process is not a mechanical positivist process. They
should use their experience and judgment to, amongst other things, interpret what constitutes a
“topic”, since it may for example be possible to divide a topic into even smaller topics. The
overriding question remains “what is it that this assessment task requires the students to process
cognitively?” This researcher’s experience dictates that a collection of ILOs, as found in a study
99
unit (SU) would most likely constitute a topic, which can be taken as an element of material
cognitive demand. The word “material” is understood as something that makes a difference, i.e.
something exceeding the trivial.
Linking these elements (concepts and topics) to the SOLO taxonomy can then be done as follows:
I. If an assessment task requires and awards an irrelevant response it is set as being on the
pre-structural level. Although it would be surprising to find such an assessment task,
examiners may face the hidden risk that an assessment task which is not aligned to any
stated or implied ILO may be seen as pre-structural. Such an assessment task will not be
seen as credible or fair.
II. If an assessment task only requires the cognitive processing of any one element (i.e.
fundamental concept or topic), it is set as being one-dimensional and on the unistructural
level.
III. If an assessment task requires the material cognitive processing of two or more elements,
but does not require relational activity between these elements, the assessment task is
set as being two-dimensional or on the multi-structural level.
IV. If an assessment task requires the material cognitive processing of two or more elements
and also requires material relational activity between these elements, the assessment task
is set as being three-dimensional or on the relational level. “Material relational activity”
means that the assessment task requires the integration of detail into a structural pattern,
as opposed to requiring the mere increase in quantity of detail.
V. If an assessment task that requires the material cognitive processing of i) two or more
elements as well as ii) material relational activity between these elements, but also iii) the
conceptualisation of responses to go beyond the given information as well as go beyond
what can reasonably and contextually be expected as at least partly conventional wisdom,
then the assessment task is set as being four-dimensional or on the extended abstract
level.
One may ask why the term “dimensional” is used above. The reason is, it was found that it
becomes easier to think of assessment tasks in terms of dimensions (2D vs 3D vs 4D) if one
refers to Biggs and Tang’s (2007:79) own drawing (see Figure 2-3, page 44). A three-dimensional
question has “depth” (a relational aspect) and a four-dimensional question ventures into the
unknown (“fourth dimension”). Being pragmatic, this described method also needed a name (as
do most things of value in life) hence the “dimensional method” was so named by this researcher.
It is furthermore noted that the utilisation of verbs should not be totally discarded, since they may
still assist, for example, in whether to decide which fundamental concepts of accounting are
required by an assessment task.
100
Similar to the current practice of universities to link assessment tasks to the revised Bloom’s
taxonomy by using the number of marks (NWU, 2011:1 and CUT, 2016:57 as examples), this
“dimensional method” also uses the number of marks for analysis purposes, where one stated
assessment task actually implies different SOLO levels.
As explained in paragraph 3.3, an abductive approach was followed for the development and
testing of this “dimensional method”. The following examples of assessment tasks from the
universities’ examinations are given as the result of coding with the SOLO taxonomy (priori
codes), as well as codes derived from the data by the researcher (i.e. the types of assessment
tasks). The examples illustrate the application of the “dimensional method”.
4.3.5.1 Example 1: Trial balance with adjustments
Examination: University A
Brief description of the scenario: The question provides a trial balance and lists ten points of
additional information (including adjustments needed). The business has the form of a sole
proprietorship. There can be an argument to name this type of question “Financial reporting”, but
the information given (trial balance and information) can also be used to require journal entries or
general ledger accounts; therefore, the example is named according to the given information of
the question. The overall “required” part is divided into two assessment tasks, of which number 1
is discussed here.
Assessment task: “Prepare the statement of profit or loss and other comprehensive income for
the year ended 31 December 20XX.” This question counts 14 marks.
Analysis through SOLO: Since the assessment task requires the adjustment of the given
information (“summarising and classifying”) as well as the preparation of a financial statement
(“presentation and disclosure”) dealing mainly with the topics of income/expenses and assets
(depreciation calculations), it can be seen that at least three elements are present. Since all 14
marks are awarded for amounts that needed adjustment, the initial reaction would be to consider
the whole task as being on the relational level, since adjustments need to be related to the
amounts. However, of these adjustments, eight marks i) does not require material relational
activity, since they comprise of simple straightforward calculations (e.g. 65 500 + 7 800) which is
ii) considered conventional wisdom in the context of Financial Accounting 1. Therefore, the
assessment task is set as having six marks (43%) on the relational SOLO level (or 3D level) and
eight marks (57%) on the multi-structural SOLO level (or 2D level). The universities will have to
decide in light of the NQF level (i.e. NQF 5 here) what the appropriate percentage for each SOLO
level is (this consideration is beyond the scope of this study).
101
Variations and other occurrences: The “Trial balance and adjustment” type question was found
to be most prevalent, including those with minor variations, e.g. requiring a different financial
statement (Universities D and F) or having a different business form (Universities A, B, C, D, F),
or requiring accounting records such as the general ledger or journal entries (University F). The
other business forms were found to be partnerships (Universities A, B, C, D) and clubs
(Universities C and F). The business form of companies was found to rather fit into example 2,
which is given separately since it is considered to be a major variation of this example. Some
examinations had more than one of these types of questions. The analysis process would
however follow the same logic as this example.
4.3.5.2 Example 2: Trial balance with adjustments and material integration of topics
Examination: University E
Brief description of the scenario: The question provides an extract from a trial balance and lists
20 points of additional information. The additional information covers basic adjustments needed,
but also provides significant detail about various topics such as share capital, debentures, taxation
and dividends. The business has the form of a company. The overall “required” part is divided
into four assessment tasks, of which number 2 is discussed here.
Assessment task: “Prepare the statement of profit or loss and other comprehensive income for
the year ended 30 September 20XX, starting from the profit before tax and finance costs figure
which was calculated in 1 above.” This question counts 13 marks.
Analysis through SOLO: The assessment task clearly comprises more than two elements
(similar to example 1) as well as some relational aspects. There is, however, no requirement to
address anything clearly beyond i) the given information together with ii) beyond what can
reasonably and contextually be expected as at least partly conventional wisdom. Therefore, the
assessment task cannot be set on the extended abstract level. The calculation of the finance
costs is a calculation that is not straight forward conventional wisdom, since the concept time
value of money is involved (this attributed six marks). The remaining summarising of the given
information into the financial statement is either done directly from the given information, or is
based on straight forward basic calculations (this represents seven marks). Therefore, the
assessment task is set as having six marks (46%) on the relational SOLO level (or 3D level) and
seven marks (54%) on the multi-structural SOLO level (or 2D level). Due to the relative length of
the given information as well as a layout that seems different to those seen in example 1 above,
the examiners and moderators should consider whether “time pressure” and “layout” does not
raise at least some of the marks to a higher SOLO level. This will depend on the “conventional
102
wisdom” for this program’s students, which as experience dictates should be fairly high (i.e. they
would have seen such a layout and question before). This would then mean that the set SOLO
levels are not disrupted.
Variations and other occurrence: University G had a similar type of question with the variation
that the statement of changes in equity was required, while University B required both the
statement of profit or loss and other comprehensive income as well as the statement of changes
in equity.
4.3.5.3 Example 3: Disclosure of notes to the financial statements
Examination: University C
Brief description of the scenario: The question provides a statement of financial position and
lists eight points of additional information. The additional information covers adjustments needed,
but also provides detail about the topics included in a statement of financial position such as
assets, equity and liabilities. The detail is given to facilitate disclosure. The question states (as
can be seen) that the given statement of financial position is not correctly prepared in terms of
IFRS. The business has the form of a company. The overall “required” part is divided into four
assessment tasks, of which number 1 is discussed here.
Assessment task: “Use the attached answer sheet and prepare the note for property, plant and
equipment as at 28 February 20XX.” This question counts 12 marks.
Analysis through SOLO: The assessment task clearly comprises at least two elements (i.e.
classification of assets and also disclosure, since although the overall format of the note is given,
the students will still have to disclose in the correct place) as well as some relational aspects,
since the additional information will have to be related to the statement of financial position in
order to complete the assessment task. There is, however, no requirement to address anything
clearly beyond: i) the given information together with ii) what can reasonably and contextually be
expected as at least partly conventional wisdom. Therefore, the assessment task cannot be set
on the extended abstract level. The calculations required by the assessment task is straight
forward and does not include for example an asset-disposal. Therefore, there is no material
relational activities and the question should be conventional wisdom for most students. This
assessment task is therefore set as being 100% (12 marks) on the multi-structural or 2D level.
Variations and other occurrences: This type of question is also quite prevalent (e.g. also seen
with Universities A, C, D, F, G) with the main variation being the requirement of different notes to
the financial statements (such as University G requiring the note for “Long term borrowings”). In
103
most cases this type of assessment task tended to be multi-structural for the same reasons as
noted above (i.e. straight forward calculations or plain transfer of information).
4.3.5.4 Example 4: Unistructural question
Examination: University C
Brief description of the scenario: This assessment task does not require any processing of the
given information although it is linked to the given information. It requires the recall from memory
of certain facts. The overall “required” part is divided into four assessment tasks, of which number
3 is discussed here.
Assessment task: “Number 4 under additional information implies that the investments must
have been revalued. Indicate the methods used to measure the value of listed and unlisted
investments respectively.” This question counts two marks.
Analysis through SOLO: The suggested solution being “market value” and “director’s value”
shows that this should be conventional wisdom for a Financial Accounting 1 student. Since the
assessment task comprises only one element, namely the topic of value (at basic level), it is set
as unistructural or 1D. Due to various reasons, some students may have found the task to be
extended abstract due to gaps in their background knowledge. This would however be unlikely
and the number or marks does not warrant concern. Unistructural assessment tasks should not
summarily be dismissed as inappropriate (see paragraph 2.5.1.3).
Variations and other occurrences: None were found, but this example illustrates the nature of
such an assessment task.
4.3.5.5 Example 5: Theory and apply type question
Examination: University G
Brief description of the scenario: The question from University G provides half a page of
information (i.e. the “case study”) about a company that manufactures plastic bottles and sells
them to various water retailers. The overall “required” part is divided into four assessment tasks,
of which number 1 is discussed here.
Assessment task: “With reference only to the recognition criteria of a provision, discuss why a
provision should be recognised by G (Pty) Ltd on 31 December 20XX in respect of the summons
from Z (Pty) Ltd’s lawyers.” This question counts six marks.
104
Analysis through SOLO: This assessment task is similar in nature to the example discussed in
paragraph 4.3.3, since this is a classic Financial Accounting 1 assessment task which entails the
listing of rules and the relating or applying of given information to those rules while ultimately
concluding with an answer. The assessment task clearly comprises at least two elements, i.e.
recognising and recording of one standalone topic, which in this case is provisions. The
aforementioned rules out the pre-structural and unistructural levels from further consideration.
The assessment task also requires some relational aspect, since the additional information will
have to be related to the stated rule in order to complete the assessment task. However, there is
no requirement to address anything clearly beyond: i) the given information together with ii)
beyond what can reasonably and contextually be expected as at least partly conventional wisdom.
Therefore, the assessment task cannot be set on the extended abstract level. The assessment
task will then in essence be either multi-structural, relational, or a combination of these. Similar to
the explanation given in paragraph 4.3.3, the task is thus seen as multi-structural if a student has
for example seen it before or have practiced it or have been guided on a similar task, which implies
that the student will list the rules (recognition criteria) and scan the given information for typical
phrases to indicate compliance or non-compliance with the rules. In such an instance, no material
relational activity was required. In terms of this example’s specific assessment task (about
provisions) the nature of the topic does not facilitate an “auto-pilot” response of finding typical
phrases to apply easily. The reason is that the provision revolves around possible litigation and
unique facts and circumstances in the given information, which will have to be carefully
considered and related to the rules in order to complete the assessment task. The six marks
(100%) are therefore suggested to be set as being on the relational SOLO level or 3D level. This
level may be materially disrupted (by the theme of conventional wisdom) to a multi-structural level
if students have seen a case study that was significantly similar (to the point of duplication) in
detail to this one (which should be unlikely). On the other hand, if a student has for some reason
missed the IAS 37 Provisions, Contingent Liabilities and Contingent Assets lecture and had no
other effective facilitation or study guidance, then the whole assessment task may become
extended abstractionism for that student.
Variations and other occurrences: Similar assessment tasks were found with Universities B,
D, E and G. University B required that the students must criticise a statement of financial position
in terms of IAS 1 Presentation of Financial Statements, which, due to the fact that students would
have to consider the unique issue in the given information (as opposed to an “auto-pilot”
response), places the task on the relational level. University D had the minor variation that the
application of the definition of a liability (instead of a provision) was required. The topic of
refunding moneys received for shares yet issued placed the task on a relational level. University
E required the application of the qualitative characteristics of the Conceptual Framework of IFRS,
105
which would also be set on the relational level, due to the specific fact and circumstances of the
case study. University G’s other example is discussed in paragraph 4.3.3.
4.3.5.6 Example 6: Cash flow statement
Examination: University B
Brief description of the scenario: The question from University B provides a statement of
financial position, which includes comparative figures. Five points of additional information
regarding items with a possible cash flow effect are also provided. The overall “required” part is
divided into four assessment tasks, of which number 1 is discussed here.
Assessment task: “Prepare the cash flows from operating activities section of B Ltd’s statement
of cash flows for the year ended 30 June 20XX, by applying the direct method.” This question
counts 15 marks.
Analysis through SOLO: This assessment clearly requires more than two elements. Firstly, it
requires the application of the concepts of classification of amounts and presentation in a financial
statement (i.e. the cash flow statement). Secondly, topics involved include the cash flow effect of
assets, liabilities, income and expenses, as guided by IAS 7 Statement of Cash Flows. There is
clearly some relational activity required in order to use the given information to complete the
assessment task, while there is (similar to the reasons stated in examples 1-5 above) no
requirement of extended abstractionism. Based on the straight forward calculations required as
well as the straight forward use of the information given, this assessment task does not require
material relational activity and can be set on the multi-structural level (100%). However,
experience shows that the cash flow statement is highly susceptible to disruption by the themes
of “time allowed” and “conventional wisdom”. The reason for time pressure includes that the cash
flow statement deals with various topics and calculations (not only two or three) as well as that
the framework for a cash flow statement takes relatively (to other tasks) more time to draw up.
This may well be one of the reasons why the cash flow statement is divided into smaller
assessment tasks, as was the case here. With practise and preparation, students should be able
to overcome this time pressure easily. In terms of conventional wisdom, students that are not
well-practiced in setting up a cash flow statement will experience a higher cognitive demand,
since they will have to relate each piece of information materially to figure out where it belongs.
For them, all of the marks will become relational or 3D in level. Examiners are therefore
encouraged to ensure that students are well-prepared for a cash flow question, in which case the
multi-structural nature should lead to good grades for this pertinent assessment task.
106
Variations and other occurrences: The cash flow statement is also prevalent and was for
example found in the examinations of Universities A, D, E and G. There was very little variation
between the examinations in terms of the type of information given, which makes sense in light
of the inherent nature of the cash flow statement’s requirements. Universities E and G did require
more calculations (more topics present) and general ledger accounts, but then again also
awarded more marks and time.
4.3.5.7 Example 7: Bank reconciliation
Examination: University F
Brief description of the scenario: The question from University F provides half a page of
information about the entity’s accounting records regarding bank and cash, but also about the
real bank’s accounting records about the entity. The overall “required” part is divided into three
assessment tasks, of which all three is given here since they will be completed simultaneously
due to the nature of a bank reconciliation.
Assessment tasks: “(a) Complete only the detail and bank columns of the cash receipts and
cash payments journals of F-Traders, for May 20XX; (b) Prepare the bank account in the general
ledger of F-Traders, properly balanced, on 31 May 20XX; (c) Prepare the bank reconciliation
statement of F-Traders as at 31 May 20XX. Begin with the balance as per the bank statement.”
This question counts 18 marks in total.
Analysis through SOLO: This assessment task clearly requires more than two elements, since
the (concept of) recognition and measurement (of the topic) of “bank and cash” is in essence
assessed. Similar to the reasons given in examples 1-5, there is no requirement for extended
abstractionism. Due to conventional wisdom (e.g. the assessment task is also comprehensively
facilitated in secondary school), there is no material relational activities required. All 18 marks
(100%) of the assessment task is therefore set as being on the multi-structural (or 2D) level.
Variations and other occurrences: Even though University F’s was the only examination found
to require a bank reconciliation for this study, experience dictates that it is quite prevalent.
Reasons for its absence may well be that the other universities assess it at other stages during
the year or assess it as part of formative assessment. The fact that conventional wisdom around
bank reconciliations tend to be quite high amongst accounting students (i.e. it is comprehensively
facilitated in secondary school) may also decrease the need for the assessment thereof.
Variations usually involve the altering of the layout of the information given. University F’s debtor
reconciliation question and University E’s creditor reconciliation question would follow the same
logic as this example.
107
4.3.5.8 Example 8: Financial ratios
Examination: University D
Brief description of the scenario: About half a page of information is given, which includes
extracts form the financial statements. The overall “required” part is divided into five assessment
tasks, of which number 3 is discussed here.
Assessment task 3: “Calculate the following for the 20XX year: 3.1 Days inventory on hand;
3.2 Creditors payment period; 3.3 Debtors collection period.” This question counts seven marks
in total.
Analysis through SOLO: A “financial ratios” question is in essence a form of the “theory and
apply” type of question and therefore follows the same logic as given in example 5 (paragraph
4.3.5.5). Briefly summarised it means that this assessment task covers at least two elements, i.e.
the i) interpretation of ii) financial statements, but does not require material relational activity since
the required response is mostly “auto-pilot” in nature, due to conventional wisdom. The task is
thus set as being multi-structural (2D) with the possibility of becoming relational should an “exotic”
case study (e.g. for Financial Accounting 1 students – a group with subsidiaries), scenario or ratio
be required, which may then even go into an extended abstract level.
Variations and other occurrences: Although University D’s was the only examination found to
require financial ratios for this study, experience dictates that it is quite prevalent. The reason for
its absence mirrors example 7’s (paragraph 4.3.5.7) reasons.
4.3.5.9 About journal entries and general ledger accounts
Journal entries and general ledger accounts are accounting activities (typically involving
recognition and measurement) that are a means to an end. The “end” being the financial
statements. Therefore, the same logic as the examples above would apply to evaluating the
SOLO level of an assessment task requiring journal entries of general ledger accounts. Examples
1, 2, 3 and 5 are especially geared toward setting journal entries or general ledger accounts as
assessment tasks. Journal entries were required in the examination of Universities A, B, D and
G, while general ledger accounts were required by Universities A, D and F. These tasks were
without exception part of an overall assessment task that was addressed in the examples above.
4.3.6 Reflecting on Theme 4
As explained in paragraphs 3.4.4.4, qualitative data analysis is mostly a process of analysing and
re-analysing data (instead of being a linear process), since while one codes, new understandings
108
emerge from the data which may alter the codes themselves or the relationships between the
different pieces of data. In light of this it was noted that the type of examples above originated
from codes that were derived from the data by the researcher (see Figure 3-3, page 75), while it
was ensured that each of the seven universities are represented by an example. In addition to
the analysis given under Theme 4 above, the following observations were also made:
No instances of pre-structural questions were found at all. This is to be expected in a professional
environment such as an accounting school, but is still gratifying to see. Very few unistructural
questions were found, and if they were present, they did not count many marks. Unistructural
questions may however have been used to balance the level of the paper or time pressure. No
examples of extended abstractionism questions were seen, which also makes sense in light of
the fact that the examinations were pitched at NQF 5 (introductory level). The questions were
found to generally be pitched at around +-60% on the multi-structural level, with the majority of
the remaining marks being on the relational level. None of the examinations covered, for example
consolidated group financial statements, which is also in line with the researcher’s experience of
the typical financial accounting ILOs at NQF 5. All in all, the examinations were perceived to be
professionally prepared and aligned to the typical financial accounting ILOs.
In time and by practising the “dimensional method”, examiners will likely be able to quickly state
an assessment task as being 1D/2D/3D or 4D in nature.
4.4 SUMMARY
In this chapter the study’s data was analysed and findings were given through thick description.
The empirical evidence generated was documented so as to show that the research objectives
as stated in paragraph 4.1 were achieved.
The results of the data analysis showed that the fundamental concepts of financial accounting
(such as recognition, measurement, etc.) as well as the applicable topics (as driven by the
intended learning outcomes and typically including assets, liabilities, income, expenses, etc.) can
be used as a proxy for the cognitive demand induced by the assessment tasks. The combination
of these concepts and topics, together with the requirement (or lack thereof) of material relational
activity or extended abstractionism, can then be used to set a SOLO level for the assessment
task. The SOLO level of an assessment task may also be materially disrupted (or even set) by
the themes of “time allowed”, “layout” or “conventional wisdom”. A framework, which includes the
examples (given in paragraph 4.3.5) of assessment tasks as evaluated through the SOLO
taxonomy, will be given in Chapter 5 as the final product of the research objectives being
achieved.
109
CHAPTER 5 CONCLUSIONS AND RECOMMENDATIONS
5.1 INTRODUCTION
This chapter includes the study’s conclusions and recommendations. It will also clearly link the
findings to the problem statement and ensure that every objective has been achieved. The final
suggested coherent framework will be given. Areas for further research and reflective learning
will also be briefly discussed.
5.2 CONFIRMATION OF THE RESEARCH OBJECTIVES ACHIEVED
An iterative process was followed in addressing the study’s research problem (stated in paragraph
1.2). This was possible as the research problem was also reformulated as research questions (in
paragraph 1.2), which in turn were reformulated into a primary research objective (paragraph
1.3.1) which in turn was broken up into secondary research objectives. Therefore, by addressing
the secondary research objectives, it can be seen that the primary objective is achieved and the
research questions and problem addressed.
Although the study as a coherent whole contributed to achieving the research objectives, Table
5-1 indicates the specific points where the various secondary objectives (as stated in paragraph
1.3.2) have been considered to be fully achieved:
Table 5-1: Research objectives achieved
Number Objective Achieved in
1 To contextualise from the literature the significant matters
surrounding summative assessment including the matters of
stakeholders’ interest and principles that should be adhered to
when setting and delivering assessments.
Chapter 2 with specifically
paragraph 2.3 (stakeholders)
and paragraph 2.4
(assessment).
2 To conceptualise the application of the SOLO taxonomy in
evaluating the standard or level of summative assessments.
Chapter 2 with specifically
paragraph 2.5 (SOLO).
3 To document as part of the proposed framework, typical
Financial Accounting 1 summative assessment tasks with the
analysis through the SOLO taxonomy.
Chapter 4 with specifically
paragraph 4.3.5, including its
sub-paragraphs.
4 To determine by analysis through the SOLO taxonomy, typical
themes that have an influence on the standard or level of a
Financial Accounting 1 summative assessment.
Chapter 4 with specifically
paragraphs 4.3.1 to 4.3.5.
110
5 To synthesise the insight gained through the study into a
framework which can guide the application of the SOLO
taxonomy in evaluating the standard of summative assessment
of a Financial Accounting 1 module.
Chapters 4 and 5 with
specifically paragraph 5.3
Source: Researcher
5.3 FRAMEWORK FOR THE APPLICATION OF THE SOLO TAXONOMY IN EVALUATING
THE LEVEL OF INTRODUCTORY FINANCIAL ACCOUNTING ASSESSMENTS
5.3.1 Introduction
The aim of this framework is to provide guidance for the application of the SOLO taxonomy in
evaluating the standard of a summative assessment of a Financial Accounting 1 (introductory
financial accounting) module.
The framework therefore also addresses the specific issue that arises from the nature of a
Financial Accounting 1 examination (especially in the context of summative assessment), where
it may be no easy task to evaluate the assessment to determine a level of cognitive demand
required, even with the aid of the revised Bloom’s taxonomy. The major reason for this being that
for assessment of Financial Accounting 1, verbs indicating assessment tasks, such as
“remember”, “apply” or “evaluate” which are usually associated with the revised Bloom’s
taxonomy, may imply different cognitive demands within a context that can also differ significantly
for each assessment task, even if the same verb occurs.
The framework is informed by research with the main aim of having practical utility. The framework
is not meant to be prescriptive, but rather to illuminate or assist. Examiners and moderators are
encouraged to use their experience and judgment to consider contextual issues and to not merely
follow a mechanical process.
The sections that follow each addresses a significant specific theme of guidance.
5.3.2 Time allowed to complete assessment tasks
Time pressure has an impact on a student’s cognitive response, and therefore examiners must
carefully contemplate whether an assessment task is to test time management skills, reading
skills, planning skills, the concepts or financial accounting, or all of these as well as to what
degree. Examiners must be careful that time pressure does not inadvertently frustrate the SOLO
level which the assessment task aims the students to operate at.
111
5.3.3 The layout of the assessment
A layout that students perceive as unfamiliar or complex can have the effect of raising the SOLO
level (which may be desired or not) of the assessment task for that student, since the effect of the
layout of the examination in itself then becomes a concept to cognitively process along with the
other requirements of the assessment task. A complex or unfamiliar layout will also affect the time
allowed to complete an assessment, since such a layout will require more time of the student in
order to function on the relational SOLO level (i.e. more “thinking time” is required).
5.3.4 The effect of “conventional wisdom”
The genesis of this theme is the notion that “today’s extended abstract is tomorrow’s relational”
(Biggs & Tang, 2007:78) where an initial breakthrough in time becomes “conventional wisdom”.
An example in terms of financial accounting is the impact of IFRS 15, which initially could be
considered as a higher-level assessment task due to the novelty thereof.
This theme also connects to the teaching strategy of building on prior knowledge (“working from
known to unknown” or “scaffolding”), wherein it is widely accepted that building on quality prior
knowledge positively influences i) further knowledge acquisition and ii) the capacity to apply
higher-order cognitive problem-solving skills.
It should however be noted that this theme will admittedly (as often found in real life) involve a
degree of professional judgment by the examiner, since conventional wisdom or the lack thereof
may raise or lower an assessment task’s SOLO level.
Examiners, in an environment of constructive alignment and fair assessment, may for example
rightfully evaluate that a particular assessment task is on a multi-structural or relational SOLO
level, but the risk is that these conditions (i.e. effective facilitation, guidance, preparation, etc.)
may be absent for a certain student. This could have the effect that the SOLO level of that
assessment task then becomes one of extended abstractionism for that student.
The recommendation of this theme is therefore that it is suggested that an examiner should, while
regarding the best available information, consider how much conventional wisdom a well-
prepared student (i.e. the “reasonable man” test) should possess in terms of a certain assessment
task. The examiner should then consider the effect of the conventional wisdom on the SOLO level
of an assessment.
112
5.3.5 The link to a more structured and formalised way of applying SOLO to the
Financial Accounting 1 examinations’ assessment tasks
A shared characteristic between the themes of “time allowed”, “layout” and “conventional wisdom”
is that they are more likely to change (or “disrupt”) a SOLO level of an assessment task than to
set it. They can sometimes help to set a SOLO level, for example where time pressure or layout
is an intended learning outcome to be assessed in itself, or where students are purposely required
to function on an extend abstract level through the design of an assessment that is clearly beyond
“conventional wisdom”. However, in general the implication is that in most cases a SOLO level
will be set (using the “dimensional method” given below), after which the impact of these “disruptor
themes” should be considered. The dimensional method that follows provides a more formalised
and structured way to apply the SOLO taxonomy to the assessment tasks of Financial Accounting
1.
5.3.6 The “dimensional method” of applying SOLO to the Financial Accounting 1
examinations’ assessment tasks
This method utilises the concept of “elements” which are to be used as a proxy of cognitive
demand in the same way as “verbs” would traditionally be used under the revised Bloom’s or
SOLO taxonomies. In terms of Financial Accounting 1 (i.e. financial accounting on NQF 5) the
elements to consider were found by asking “what is it that this assessment task requires the
students to process cognitively?”
The main elements were found to be:
The fundamental concepts of accounting (see paragraph 4.2) that an assessment task
requires (e.g. recognition, measurement, classification, presentation or interpretation,
etc.).
The topics of financial accounting that are involved, e.g. bank and cash or inventory etc.
These topics would for example be derived from the ILOs which constitute the curriculum.
Examiners should note that this process is not a mechanical positivist process. They should use
their experience and judgment to, amongst other things, interpret what constitutes a “topic”, since
it may for example be possible to divide a topic into even smaller topics. The overriding question
remains “what is it that this assessment task requires the students to process cognitively?” This
researcher’s experience dictates that a collection of ILOs as found in a study unit (SU) would most
likely constitute a topic which can be taken as an element of material cognitive demand. The word
“material” is understood as something that makes a difference, i.e. something exceeding the
trivial.
113
Linking these elements (concepts and topics) to the SOLO taxonomy can then be done as follows:
I. If an assessment task requires and awards an irrelevant response, it is set as being on
the pre-structural level. Although it would be surprising to find such an assessment task,
examiners may face the hidden risk that an assessment task that is not aligned to any
stated or implied ILO may be seen as pre-structural. Such an assessment task will not be
seen as credible or fair.
II. If an assessment task only requires the cognitive processing of any one element (i.e.
fundamental concept or topic), it is set as being one-dimensional and on the unistructural
level.
III. If an assessment task requires the material cognitive processing of two or more elements
but does not require relational activity between these elements, the assessment task is
set as being two-dimensional or on the multi-structural level.
IV. If an assessment task requires the material cognitive processing of two or more elements
and also requires material relational activity between these elements, the assessment task
is set as being three-dimensional or on the relational level. “Material relational activity”
means that the assessment task requires the integration of detail into a structural pattern
as opposed to requiring the mere increase in quantity of detail.
V. If an assessment task that requires the material cognitive processing of i) two or more
elements as well as ii) requires material relational activity between these elements, but
also requires iii) the conceptualisation of responses to go beyond the given information as
well as go beyond what can reasonably and contextually be expected as at least partly
conventional wisdom, then the assessment task is set as being four-dimensional or on the
extended abstractionism level.
The word “dimensional” is used as a way to remember the method and links the method to Biggs
and Tang’s (2007:79) pertinent visualisation of the SOLO taxonomy (given here as part of Figure
5-1, page 115). A three-dimensional question has “depth” (a relational aspect) and a four-
dimensional question ventures into the unknown (“fourth dimension”). Examiners will likely in time
and by practising the “dimensional method” be able to quickly state an assessment task as being
1D/2D/3D or 4D in nature.
Similar to the current practice of universities to link assessment tasks to the revised Bloom’s
taxonomy by using the number of marks, the dimensional method can also use the number of
marks for analysis purposes, where one stated assessment task actually implies different sub-
tasks and SOLO levels.
114
It is furthermore noted that the utilisation of verbs should not be totally discarded, since they may
still assist in, for example, to decide which fundamental concepts of accounting are required by
an assessment task.
Table 5-2 (page 115) provides examples of typical Financial Accounting 1 assessment tasks with
the application of the dimensional method.
Figure 5-1: A visual summary of the SOLO taxonomy and assessment
Source: Adapted from Biggs and Tang (2007:79) and Hook (2015:5)
115
Table 5-2: Examples of typical Financial Accounting 1 assessment tasks with the application of the dimensional method
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
Example 1:
Trial balance
with
adjustments
The question provides a
trial balance and lists ten
points of additional
information (including
adjustments needed). The
business has the form of a
sole proprietorship. There
can be an argument to
name this type of question
“Financial reporting”, but
the information given (trial
balance with further
information) can also be
used to require journal
entries or general ledger
accounts; therefore the
example is named
according to the given
information of the
question.
Prepare the statement
of profit or loss and
other comprehensive
income for the year
ended 31 December
20XX.
This question counts 14
marks.
Since the assessment task requires the
adjustment of the given information
(“summarising and classifying”) as well as the
preparation of a financial statement
(“presentation and disclosure”) dealing mainly
with the topics of income/expenses and assets
(depreciation calculations), it can be seen that
at least three elements are present. Since all 14
marks are awarded for amounts that needed
adjustment, the initial reaction would be to
consider the whole task as being on the
relational level, since adjustments need to be
related to the amounts. However, of these
adjustments, eight marks i) do not require
material relational activity since they comprise
of simple straightforward calculations (e.g.
65 500 + 7 800) which is ii) considered
conventional wisdom in the context of Financial
Accounting 1. Therefore, the assessment task is
set as having six marks (43%) on the relational
SOLO level (or 3D level) and eight marks (57%)
on the multi-structural SOLO level (or 2D level).
The universities will have to decide in light of the
The “Trial balance and
adjustment” type question was
found to be most prevalent,
including those with minor
variations, e.g. requiring a
different financial statement or
having a different business form
or requiring accounting records
such as the general ledger or
journal entries. The other
business forms were found to
be partnerships and clubs. The
business form of companies
was found to rather fit into
example 2, which is given
separately, since it is
considered to be a major
variation of this example. Some
examinations had more than
one of these types of questions.
The analysis process would
however follow the same logic
as this example.
116
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
NQF level (i.e. NQF 5) what the appropriate
percentage for each SOLO level is.
Example 2:
Trial balance
with
adjustments
and material
integration of
topics
The question provides an
extract from a trial balance
and lists 20 points of
additional information. The
additional information
covers basic adjustments
needed, but also provides
significant detail about
various topics such as
share capital, debentures,
taxation and dividends.
The business has the form
of a company.
Prepare the statement
of profit or loss and
other comprehensive
income for the year
ended 30 September
20XX, starting from the
profit before tax and
finance costs figure
which was calculated in
1 above.
This question counts 13
marks.
The assessment task clearly comprises more
than two elements (similar to example 1) as well
as some relational aspects. There is however
no requirement to address anything clearly
beyond i) the given information together with ii)
what can reasonably and contextually be
expected as at least partly conventional
wisdom. Therefore, the assessment task cannot
be set on the extended abstract level. The
calculation of the finance costs is a calculation
that is not straight forward conventional wisdom,
since the concept time value of money is
involved (this attributed six marks). The
remaining summarising of the given information
into the financial statement is either done
directly from the given information or is based
on straight forward basic calculations (this
represents seven marks). Therefore, the
assessment task is set as having six marks
(46%) on the relational SOLO level (or 3D level)
and seven marks (54%) on the multi-structural
SOLO level (or 2D level). Due to the relative
One university had a similar
type of question with the
variation that the statement of
changes in equity was required,
while another university
required both the statements of
profit or loss and other
comprehensive income as well
as the statement of changes in
equity.
117
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
length of the given information as well as a
layout that seems different to those seen in
example 1 above, the examiners and
moderators should consider whether “time
pressure” and “layout” do not raise at least
some of the marks to a higher SOLO level. This
will depend on the “conventional wisdom” for
this program’s students, which, as experience
dictates, should be fairly high (i.e. they would
have seen such a layout and question before).
This would then mean the set SOLO levels are
not disrupted.
Example 3:
Disclosure of
notes to the
financial
statements
The question provides a
statement of financial
position and lists eight
points of additional
information. The additional
information covers
adjustments needed but
also provides detail about
the topics included in a
statement of financial
position such as assets,
equity and liabilities. The
Use the attached
answer sheet and
prepare the note for
property, plant and
equipment as at 28
February 20XX.
This question counts 12
marks.
The assessment task clearly comprises at least
two elements (i.e. classification of assets and
also disclosure, since although the overall
format of the note is given, the students will still
have to disclose in the correct place) as well as
some relational aspects, since the additional
information will have to be related to the
statement of financial position in order to
complete the assessment task. There is
however no requirement to address anything
clearly beyond: i) the given information together
with ii) what can reasonably and contextually be
This type of question is also
quite prevalent, with the main
variation being the requirement
of different notes to the financial
statements (such as requiring
the note for “Long term
borrowings”). In most cases this
type of assessment task tended
to be multi-structural for the
same reasons as noted above
(i.e. straight forward
118
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
detail is given to facilitate
disclosure. The question
states (as can be seen)
that the given statement of
financial position is not
correctly prepared in
terms of IFRS.
expected as at least partly conventional
wisdom. Therefore, the assessment task cannot
be set on the extended abstractionism level.
The calculations required by the assessment
task is straight forward and does not include an
asset-disposal, for example. Therefore, there is
no material relational activities and the question
should be conventional wisdom for most
students. This assessment task is therefore set
as being 100% (12 marks) on the multi-
structural or 2D level.
calculations or plain transfer of
information).
Example 4:
Unistructural
question
This assessment task
does not require any
processing of the given
information although it is
linked to the given
information. It requires the
recall from memory of
certain facts.
The additional
information implies that
the investments must
have been revalued.
Indicate the methods
used to measure the
value of listed and
unlisted investments
respectively.
This question counts
two marks.
The suggested solution being “market value”
and “director’s value” shows that this should be
conventional wisdom for a Financial Accounting
1 student. Since the assessment task comprises
only one element, namely the topic of value (at
basic level), it is set as unistructural or 1D. Due
to various reasons, some students may have
found the task to be extended abstractionism
due to gaps in their background knowledge.
This would however be unlikely and the number
or marks does not warrant concern.
Unistructural assessment tasks should not
summarily be dismissed as inappropriate.
This example illustrates the
nature of such an assessment
task.
119
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
Example 5:
Theory and
apply type
question
The question provides half
a page of information (i.e.
the “case study”) about a
company that
manufactures plastic
bottles and sells them to
various water retailers.
With reference only to
the recognition criteria
of a provision, discuss
why a provision should
be recognised by G
(Pty) Ltd on
31 December 20XX in
respect of the summons
from Z (Pty) Ltd.’s
lawyers.
This question counts six
marks.
This assessment task is similar in nature to the
example discussed with the theme of
“conventional wisdom”, since this is a classic
Financial Accounting 1 assessment task which
entails the listing of rules and the relating or
applying of given information to those rules,
while ultimately concluding with an answer. The
assessment task clearly comprises of at least
two elements, i.e. recognising and recording of
one standalone topic which is, in this case,
provisions. The aforementioned rules out the
pre-structural and unistructural levels from
further consideration. The assessment task also
requires some relational aspect, since the
additional information will have to be related to
the stated rule in order to complete the
assessment task. There is, however, no
requirement to address anything clearly beyond:
i) the given information together with ii) what
can reasonably and contextually be expected as
at least partly conventional wisdom. Therefore,
the assessment task cannot be set on the
extended abstract level. The assessment task
will then in essence be either multi-structural,
Similar assessment tasks were
found in the examinations of the
different universities. One
university required that the
students must criticise a
statement of financial position in
terms of IAS 1 Presentation of
Financial Statements, which
due to the fact that students
would have to consider the
unique issue in the given
information (as opposed to an
“auto-pilot” response) place the
task on the relational level.
Another university had the
minor variation that the
application of the definition of a
liability (instead of a provision)
was required. The topic of
refunding moneys received for
shares yet issued placed the
task on a relational level.
Another university required the
application of the qualitative
120
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
relational or a combination of these. Similar to
the explanation given with the theme of
“conventional wisdom”, the task is thus seen as
multi-structural if a student has for example
seen it before, has practiced it or has been
guided on a similar task, which implies that the
student will list the rules (recognition criteria)
and scan the given information for typical
phrases to indicate compliance or non-
compliance with the rules. In such an instance,
no material relational activity was required. In
terms of this example’s specific assessment
task (about provisions), the nature of the topic
does not facilitate an “auto-pilot” response of
finding typical phrases to apply easily. The
reason is that the provision revolves around
possible litigation and unique facts and
circumstances in the given information, which
will have to be carefully considered and related
to the rules in order to complete the assessment
task. The six marks (100%) are therefore
suggested to be set as being on the relational
SOLO level or 3D level. This level maybe
materially disrupted (by the theme of
characteristics of the
Conceptual Framework of
IFRS, which would also be set
on the relational level due to the
specific facts and
circumstances of the case
study.
121
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
conventional wisdom) to a multi-structural level
if students have seen a case study that was
significantly similar (to the point of duplication)
in detail to this one (which should be unlikely).
On the other hand, if a student has for some
reason missed the IAS 37 Provisions,
Contingent Liabilities and Contingent Assets
lecture and had no other effective facilitation or
study guidance, then the whole assessment
task may become extended abstractionism for
that student.
Example 6:
Cash flow
statement
The question provides a
statement of financial
position that includes
comparative figures. Five
points of additional
information regarding
items with a possible cash
flow effect are also
provided.
Prepare the cash flow
from operating activities
section of B Ltd.’s
statement of cash flows
for the year ended 30
June 20XX, by applying
the direct method.
This question counts 15
marks.
This assessment clearly requires more than two
elements. Firstly, it requires the application of
the concepts of classification of amounts and
presentation in a financial statement (i.e. the
cash flow statement). Secondly, topics involved
include the cash flow effect of assets, liabilities,
income and expenses as guided by IAS 7
Statement of Cash Flows. There is clearly some
relational activity required in order to use the
given information to complete the assessment
task, while there is (similar to the reasons stated
in examples 1 to 5 above) no requirement of
extended abstractionism. Based on the straight
The cash flow statement is also
prevalent and was found in the
examinations of various
Universities. There was very
little variation between the
examinations in terms of the
type of information given, which
makes sense in light of the
inherent nature of the cash flow
statement’s requirements.
Some universities did require
more calculations (more topics
present) and general ledger
122
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
forward calculations required as well as the
straight forward use of the information given,
this assessment task does not require material
relational activity and can be set on the multi-
structural level (100%). However, experience
shows that the cash flow statement is highly
susceptible to disruption by the themes of “time
allowed” and “conventional wisdom”. The
reason for time pressure includes that the cash
flow statement deals with various topics and
calculations (not only two or three) as well as
that the framework for a cash flow statement
takes relatively (to other tasks) more time to
draw up. This may well be one of the reasons
why the cash flow statement is divided into
smaller assessment tasks as, was the case
here. With practise and preparation, students
should be able to overcome this time pressure
easily. In terms of conventional wisdom,
students that are not well-practiced setting up a
cash flow statement will experience a higher
cognitive demand, since they will have to relate
each piece of information materially to figure out
where it belongs. For them all of the marks will
accounts, but then also
awarded more marks and time.
123
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
become relational or 3D in level. Examiners are
therefore encouraged to ensure that students
are well prepared for a cash flow question, in
which case the multi-structural nature should
lead to good grades for this pertinent
assessment task.
Example 7:
Bank
reconciliation
The question provides half
a page of information
about the entity’s
accounting records
regarding bank and cash,
but also about the real
bank’s accounting records
about the entity.
(a) Complete only the
detail and bank
columns of the cash
receipts and cash
payments journals of
F-Traders, for May
20XX; (b) Prepare the
bank account in the
general ledger of F-
Traders, properly
balanced, on 31 May
20XX;
(c) Prepare the bank
reconciliation statement
of F-Traders as at 31
May 20XX. Begin with
the balance as per the
bank statement.
This assessment task clearly requires more
than two elements, since the (concept of)
recognition and measurement (of the topic) of
“bank and cash” is in essence assessed. Similar
to the reasons given in examples 1 to 5, there is
no requirement for extended abstractionism.
Due to conventional wisdom (e.g. the
assessment task is also comprehensively
facilitated in secondary school) there is no
material relational activities required. All 18
marks (100%) of the assessment task is
therefore set as being on the multi-structural (or
2D) level.
Although only one examination
was found to require a bank
reconciliation, experience
dictates that it is quite
prevalent. Reasons for its
absence may well be that the
other universities assess it at
other stages during the year or
assess it as part of formative
assessment. The fact that
conventional wisdom around
bank reconciliations tend to be
quite high amongst accounting
students (i.e. it is
comprehensively facilitated in
secondary school) may also
decrease the need for the
assessment thereof. Variations
124
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
This question counts 18
marks in total.
usually involve the altering of
the layout of the information
given. Debtor reconciliation
questions and creditor
reconciliation questions would
follow the same logic as this
example.
Example 8:
Financial ratios
About half a page of
information is given, which
includes extracts from the
financial statements.
1. Calculate the
following for the 20XX
year:
1.1 Days inventory on
hand;
1.2 Creditors’ payment
period;
1.3 Debtors’ collection
period.
This question counts
seven marks in total.
A “financial ratios” question is in essence a form
of the “theory and apply” type question and
therefore follows the same logic as given in
example 5. Briefly summarised, it means that
this assessment task covers at least two
elements, i.e. the i) interpretation of ii) financial
statements, but does not require material
relational activity, since due to conventional
wisdom the required response is mostly “auto-
pilot” in nature. The task is thus set as being
multi-structural (2D) with the possibility of
becoming relational should an “exotic” case
study (e.g. for Financial Accounting 1 students a
group with subsidiaries), scenario or ratio be
required, which may then even go into an
extended abstractionism level.
Although only one examination
was found to require financial
ratio’s, experience dictates that
it is quite prevalent. The reason
for its absence mirrors example
7’s reasons.
125
Example Brief description of the
scenario
Assessment task Analysis through SOLO Variations and occurrence
About journal
entries and
general ledger
accounts
Journal entries and general ledger accounts are accounting activities (typically
involving recognition and measurement) that are a means to an end. The “end”
being the financial statements. Therefore, the same logic as the examples above
would apply to evaluating the SOLO level of an assessment task requiring journal
entries of general ledger accounts. Examples 1, 2, 3 and 5 are especially geared
toward setting journal entries or general ledger accounts as assessment tasks.
Journal entries were required in the examinations of various universities, while
general ledger accounts were required by some universities. These tasks were
without exception part of an overall assessment task which was addressed in the
examples above
Source: Researcher
126
5.3.7 Conclusion
This framework suggests a practical way of how to evaluate the SOLO level of each assessment
task, specifically for an introductory financial accoutring summative assessment. It is then up to
the universities to determine, in consultation with the NQF and their own policies, what percentage
of their assessment should be set at each SOLO level. The framework promises to at least add a
measure of objectivity to the evaluation of an introductory financial accounting assessment.
5.4 LIMITATIONS OF THE STUDY
Although the results of the study are based on a rigorous process, they are still subject to
limitations. It is considered beneficial to the readers of the study that these limitations are
highlighted while it is also noted how these limitations were managed so as to retain overall
credibility.
The main limitations include:
The lack of generalisability of qualitative research which stems mainly from the nature of
the data being textual and not susceptible to statistical extrapolation. Furthermore, non-
probability sampling is typically found in qualitative research, such as this study’s use of
purposive sampling. Since the study’s main question is one of “how to apply SOLO”, the
rich description during data analysis leading to the ultimate framework does mean that
there is transferability of the findings to other data sets (Financial Accounting 1
examinations) as seen in Saunders et al. (2016:206).
There is a high degree of subjectivity in the data analysis, since the researcher alone
coded the data. This subjectivity, even if defended as being provided for under the
interpretive paradigm, may still dent the study’s credibility. To compensate for this
limitation, the data analysis and findings were richly and transparently described in order
to demonstrate logic and the lack of undue bias. The researcher points to his experience
to demonstrate his expertise in relation to Financial Accounting 1 examinations. The
dependability and credibility of the study is also comprehensively considered in paragraph
3.4.4.8.
The study utilises data that is a snapshot in time (i.e. for one year) which, if the nature of
financial accounting examinations change fundamentally, will render the study outdated.
Currently there is no indication that Financial Accounting 1 summative examinations will
change materially from the data that this study used and the findings will therefore be
relevant for the foreseeable future. The final framework is however also suited to future
modification.
127
The study focused on summative assessment and the findings may not necessarily be
applicable to formative assessment or alternative forms of assessment (such as student
presentations).
It is obvious that the study is very specifically focused on Financial Accounting 1 and the
findings have not been stated for other subjects or even other levels of financial
accounting.
5.5 AREAS FOR FURTHER RESEARCH
Areas for further research were identified in two ways as set out below. These two categories are
not mutually exclusive.
5.5.1 Topics noted as extending beyond the defined research objectives
Here the topics included issues such as:
the question as to the reasons why the different universities allow a certain amount of time
for completion of the assessment tasks;
whether time pressure may help some students to focus better on the task at hand, which
may then compensate for the effect of the time-stress response;
why and how the layout of an assessment was decided on;
grading (e.g. awarding of half marks) may be investigated;
the use of multiple-choice questions for the assessment of financial accounting may well
be an interesting phenomenon to pursue;
the universities will have to decide in light of the NQF level (i.e. NQF 5 in this regard) what
percentage of the assessment tasks should be set on what SOLO level.
5.5.2 Further topics that may logically build on this study
It follows that this study may be extended to the assessment of financial accounting under higher
NQF levels.
A study that compares students’ perception of the cognitive demands of a financial accounting
exam against the findings of the application of the above framework, may prove to show some
interesting perspectives.
Another topic that was discovered whilst performing the literature review of Chapter 2 is that the
NQF level descriptors (SAQA, 2012) can possibly be evaluated through the SOLO taxonomy to
determine whether a qualification’s assessment tasks and their stated NQF levels are indeed
aligned or not.
128
5.6 REFLECTIVE LEARNING
A study such as this one has numerous advantages, of which a main one is that the researcher
engages in deep learning throughout. The following points are some of the highlights of the
learning that took place:
I learned a lot about the SOLO taxonomy, assessment and constructive alignment.
Research methodology was possibly the biggest learning point.
I honed my professional writing skills and project management skills.
I saw how different universities assess their Financial Accounting 1 students.
I realised the importance of research ethics, validity and credibility throughout the process.
Above all such a project instils perseverance, patience and gratitude.
5.7 SUMMARY
In this chapter the study’s conclusions and recommendations were given. It also clearly linked the
findings to the problem statement and ensured that every objective had been achieved. The final
suggested coherent framework was given and areas for further research, and reflective learning
was briefly discussed.
129
BIBLIOGRAPHY
Ambrose, G. & Harris, P. 2005. Layout: An arrangement of parts etc. according to a plan.
Lausanne, Switzerland: AVA Publishing SA.
Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R.,
Raths, J. & Wittrock, M.C. 2001. A taxonomy for learning, teaching and assessing: A revision of
Bloom’s Taxonomy of educational objectives. Abridged edition. New York: Longman.
Biggs, J. & Tang, C. 2007. Teaching for quality learning at university. 3rd ed. New York: Society
for Research into Higher Education & Open University Press.
Biggs, J. 2019. SOLO Taxonomy. Weblog. http://www.johnbiggs.com.au/academic/solo-
taxonomy/ Date of access: 22 Jan. 2019.
Bertram, C. & Christiansen, I. 2017. Understanding research: An introduction to reading
research. 1st ed. Pretoria; Van Schaik Publishers
Burrell, G. & Morgan, G. 1979. Sociological paradigms and organisational analysis: Elements of
the sociology of corporate life. Hants, UK: Ashgate.
Bloom, B.S., Engelhart, M.D., Furst, E.J., Walker, H.H. & Krathwohl, D.R. 1956. Taxonomy of
educational objectives: The classification of educational goals handbook 1. New York:
Longmans, Green.
Bowen, G.A. 2009. Document analysis as a qualitative research method. Qualitative Research
Journal, 9(2):27-40.
Brown, G. 2001. Assessment: A guide for lecturers. York (UK): Learning and Teaching Support
Network (LTSN) Generic Centre.
https://www.researchgate.net/publication/265398364_Assessment_A_Guide_for_Lecturers
Date of access: 13 Oct. 2020.
Burke, M.E. 2007. Making choices – Research paradigms and information management:
Practical applications of philosophy in IM research. Manchester: University of Salford
Burney, S.M.A. 2008. Inductive & Deductive: Research Approach.
https://www.researchgate.net/profile/SMAqil_Burney/publication/330350434_Inductive_and_De
ductive_Research_Approach/links/5c3b2462299bf12be3c435e7/Inductive-and-Deductive-
Research-Approach.pdf Date of access: 14 Jul. 2020.
130
Bush, H.M., Daddysman, J. & Charnigo, R. 2014. Improving outcomes with Bloom’s taxonomy:
From statistics education to research partnerships. Journal of Biometrics & Biostatistics, 5(4):1-
3.
Clark, A.M. 1998. The qualitative-quantitative debate: Moving from positivism and confrontation
to post-positivism and reconciliation. Journal of Advanced Nursing, 27:1242–1249.
Clark, D. 2015. Bloom's taxonomy of learning domains.
http://www.nwlink.com/~donclark/hrd/bloom.html Date of access: 22 Oct. 2017.
Cloete, M. & Marimuthu, F. 2015. Basic accounting for non-accountants. 2nd ed. Pretoria: Van
Schaik Publishers.
Cohen, L., Manion, L. & Morrison K. 2011. Research methods in education. 7th ed. London:
Routledge.
Course Hero Inc. 2018. University of South Africa FAC 1502.
https://www.coursehero.com/sitemap/schools/3620-University-of-South-
Africa/courses/4493966-FAC1502/?__chid=3cab5cb7-fa27-4937-b044-
745ad52403a7#/documents Date of access: 12 Oct. 2020.
Creswell, J.W. 2014. Research design: Qualitative, quantitative, and mixed methods
approaches. 4th ed. CA: Sage Publications.
Creswell, J.W. 2015. Concise introduction to mixed methods research. 41st ed. CA: Sage
Publications.
CUT (Central University of Technology). 2016. Student Assessment Manual 2016.
Bloemfontein.
Dave, P. 2013. Methodological assumptions.
https://blogs.baruch.cuny.edu/com9640epstein/?p=482 Date of access: 13 Oct. 2017.
De Corte, E. 1990. Acquiring and teaching cognitive skills: A state-of-the-art of theory and
research. In Drenth, P.J.D., Sergeant, J.A. & Takens, R.J., eds. European perspectives in
psychology, Vol. 1: Theoretical, psychometrics, personality, developmental, educational,
cognitive, gerontological. John Wiley & Sons: Oxford. pp. 237-263. https://search-ebscohost-
com.nwulib.nwu.ac.za/login.aspx?direct=true&db=psyh&AN=1990-98630-014&site=eds-live
Date of access: 7 Oct. 2020.
131
Department of Higher Education and Training (South Africa). 2014. The Higher Education
qualifications sub-framework. (Notice 819). Government Gazette, 38116:10, 17 October.
De Vos, A.S, Strydom, H., Fouché, C.B. & Delport, C.S.L. 2018. Research at grass roots for the
social sciences and human service professions. 4th ed. Pretoria: Van Schaik Publishers.
Gray, D.E. 2014. Doing research in the real world. 3rd ed. London: Sage Publications.
Hart, C. 1998. Doing a literature review: releasing the social science research imagination. 1st
ed. London: Sage Publications.
Gravett, S. & Geyser, H. 2004. Teaching and learning in higher education. Pretoria, Van Schaik
Publishers.
Henning, E. 2005. Finding your way in qualitative research. Pretoria: Van Schaik Publishers.
Herbert, I., Joyce, J. & Hassall, T. 2009. Maintaining alignment in management education: The
potential for drift in assessment. International Journal of Management Education, 7(2):21-31.
Higher Education Act 101 of 1997
Hook, P. 2015. First steps with SOLO taxonomy applying the model in your classroom.
Invercargill: Essential Resources Educational Publishers Limited.
Huitt, W. 2011. Bloom et al.'s taxonomy of the cognitive domain. Educational Psychology
Interactive. Valdosta, GA: Valdosta State University.
http://www.edpsycinteractive.org/topics/cognition/bloom.html [pdf] Date of access: 22 Oct. 2017.
International Accounting Education Standards Board (IAESB). 2019. Handbook of international
education pronouncements 2019 edition IES. New York: IFAC.
International Accounting Standards Board (IASB). 2018. The conceptual framework for financial
reporting. 2018 IFRS Foundation.
http://eifrs.ifrs.org/eifrs/ViewContent?num=0&fn=Conceptual_Framework_HTML_USE_ONLY.ht
ml&collection=2018_Red_Book Date of access: 31 Oct. 2018.
International Federation of Accountants (IFAC). 2020. South African Institute of Chartered
Accountants. https://www.ifac.org/about-ifac/membership/members/south-african-institute-
chartered-accountants Date of access: 20 Oct. 2020.
132
Krathwohl, D.R. 2002. A revision of Bloom’s taxonomy: An overview. Theory into practice,
41(4):212-218.
Lucas, K.C., Dippenaar, S.M. & Du Toit, P.H. 2014. Analysis of assessment practice and
subsequent performance of third year level students in natural sciences. African education
review, 11(4):563-583.
Lucas, U. & Mladenovic, R. 2009. The identification of variation in students’ understandings of
disciplinary concepts: The application of the SOLO taxonomy within introductory accounting.
Higher Education: The International Journal of Higher Education Research, 58(2):257-283.
McLeod, S.A. 2018. Jean Piaget's theory of cognitive development.
https://www.simplypsychology.org/piaget.html Date of access: 18 Sept. 2019.
Morgan, D.L. 2014. Pragmatism as a paradigm for social research. Qualitative Inquiry,
published online 3 February 2014. Available at
http://qix.sagepub.com/content/early/2014/01/31/1077800413513733
Mouton, J. 2013. How to succeed in your Master’s and Doctoral Studies: A South African guide
and resource book. 18th impression. Pretoria: Van Schaik.
Myburgh, J.E., Fouché, J.W., Cloete, M. 2012. Accounting – An Introduction. 11th ed.
Johannesburg: LexisNexis.
Natal Joint Municipal Pension Fund v Endumeni Municipality 2012 (4) SA 593 (SCA) (16 March
2012)
National Qualifications Framework Act 67 of 2008
Newton, G. & Martin, E. 2013. Blooming, SOLO taxonomy, and phenomenography as
assessment strategies in undergraduate science education. Journal of College Science
Teaching, 43(2):78-90.
NRF (National Research Foundation). 2015. About Nexus. http://nrfnexus.nrf.ac.za/ Date of
access: 3 Oct. 2017.
NWU (North-West University). 2011. Assessment and moderation policy. Potchefstroom.
NWU (North-West University). 2012. Moderator's Report. Potchefstroom.
Rowntree, D. 1987. Assessing students: how shall we know them? London: Harper & Row.
133
SAQA (South African Qualifications Authority). 2001. Criteria and guidelines for assessment of
NQF registered unit standards and qualifications. Pretoria: SAQA.
SAQA (South African Qualifications Authority). 2012. Level descriptors for the South African
National Qualifications Framework. Pretoria: SAQA.
https://www.saqa.org.za/docs/misc/2012/level_descriptors.pdf Date of access: 12 Oct. 2020.
SAQA (South African Qualifications Authority). 2014. National policy and criteria for designing
and implementing assessment for NQF qualifications and part-qualifications and professional
designations in South Africa. Pretoria: SAQA. https://www.saqa.org.za/documents/policies-and-
criteria/national-policy-and-criteria-designing-and-implementing-assessment-nqf-qualifications-
and-part-qualifications-and-professional-designations-south-africa Date of access: 9 Oct. 2020.
SAQA (South African Qualifications Authority). 2017. Search for a registered qualification.
http://regqs.saqa.org.za/search.php Date of access: 10 Oct. 2017.
Saunders, M., Lewis, P. & Thornhill, A. 2016. Research methods for business students. 7th ed.
Harlow: Pearson.
Slack, F., Beer, M. & Armitt, G. 2003. Assessment and learning outcomes: The evaluation of
deep learning in an on-line course. Journal of Information Technology Education, 2:305-318
Smith, M. 2011. Research methods in accounting. 2nd ed. London: Sage.
Struwig, F.W. & Stead, G.B. 2001. Planning, designing and reporting research. 1st ed. Cape
Town: Pearson Education South Africa.
Tesch, R. 1990. Qualitative research: Analysis types & tools. 1st ed. London: Routledge.
Trochim, W.M.K. & Donnelly, J.P. 2008. The research methods knowledge base. 3rd ed.
Mason, OH: Cengage Learning.
UNIZULU (University of Zululand). 2015. Report by external examiner. Kwadlengezwa.
USAf (Universities South Africa). 2020. Public universities in South Africa.
http://www.usaf.ac.za/public-universities-in-south-africa/# Date of access: 20 Oct. 2020.
Van Rooyen, S. 2016. Pertinence and alignment of educational level requirements in
assessment: The case of chartered accountancy programmes in South Africa. Potchefstroom:
NWU. (Thesis – PhD).
134
Van Tonder, S.P. 2010. Assessment. Faculty of Education. Bloemfontein: University of the Free
State. (Study guide, HOS 512/FES 512)
Walliman, N. 2011. Research methods: The basics. London: Routledge.
Wallis, M. 2019. Interpretation before and after natal joint municipal pension fund v Endumeni
Municipality 2012 4 SA 593 (SCA) PER/PELJ 2019(22) – doi: http://dx.doi.org/10.17159/1727-
3781/2019/v22i0a7416
Young, B. 2013. The rhetorical philosophical assumption.
https://blogs.baruch.cuny.edu/com9640epstein/?author=14094 Date of access: 13 Oct. 2017.
135
ANNEXURES