Designing an Evaluation of a Tertiary Preparatory Program within the University Context

By Ravinesh C Deo

Presentation Overview• Introduction to a Program Evaluation?

• Benefits of an Evaluation

• Problem Statement– Challenges during First Year of Tertiary Study– Preparatory Program in Schools

• Designing an Evaluation

• Ethical Factors to Consider

• Reporting Back Mechanisms

• Conclusion– Limitations of Current Plan

Contents

What is a Program Evaluation• A process that examines success of an educational event,

syllabus design, content, implementation or achievement of objectives. assesses merits of a product, a program or a practice

• Is conducted using a multi-faceted and systematic approach focus on design, implementation and effectiveness

• Outcomes enable educational practitioners to gather evidence whether their actions and decisions are appropriate

• Helps determine whether their program fulfils aim and ethos of their academic institutions

The Tertiary Preparatory Program

• The Open Access College of University of Southern Queensland designed a teaching and a learning program known as the “Preparatory Program in Schools”.

designed for Year 11 students to complete a set of tertiary courses during their final school years

fulfils commitment of the College to supporting students who are not able to make informed decisions about career path

creates avenues for various career options

Why Evaluate the Tertiary Preparatory Program in Schools?• A new program so a need to assess the educational quality

• An opportunity to identify components that require improvement

• Outcomes may be relevant to the success of the program in areas like policy development

decision-making

funding opportunities

partnerships between the institution and community.

• Promote confidence that academic programs are being monitored, reviewed, modified and researched

• Identify challenges faced by first year students in adapting to university teaching and learning environments

Challenges Faced by First Year Students• School leavers face challenges during first year studies, as

evidenced by

Nationally attrition rate of 28.6% in 2002

Queensland attrition rate of 31.1%

After first year, a drop to 10 % (national) & 11.5 % in QLD(DETA, 2004).

• Peel (1999) identified some challenges as

inadequate preparation for independent style of learning

lack of prior exposure to tertiary environments

gaps between course structures and expectations of university and school

“expectation” that university will deliver life challenges

diversity of teacher-student relationship and a more formal relationship with lecturers

Performance Linked to Lack of Motivation?

Research Evidence? Lowe and Cook (2003) showed 20% of students could not

adjust to academic and social demands of a university.

Hillman (2005) found that students from low socio-economic backgrounds have difficulties in balancing work & study commitments.

• These could be worsened by poor motivation and lead to a diminishing of interest in studying

• As a result –performance during first year at university becomes poor.

The Government’s Decision• The Bradley Review examined our higher education

sector. incorporated recommendations on encouraging

participation of persons from low socio economic backgrounds in higher education.

focussed on enhancing partnerships between schools, institutions and communities through outreach activities.

• Declared that by 2020, 20% of tertiary enrolments are expected to be from low socio-economic regions

(Bradley, Noonan, Nugent, & Scales, 2008).

Response of Open Access College

• Committed to social inclusion for supporting students who are unable to make informed decisions about prospective careers.

• Provided pathways to those who miss on education opportunities Preparatory Program in Schools

Created alternative pathway for students to complete two tertiary level courses during final school years.

• Designed to increase participation from Ipswich & Moreton region identified as Socio Economic Disadvantaged

Only 35.7% and 19.4% respectively, of Year 12 students enter tertiary institutions (DETA, 2008; Australian Bureau of Statistics, 2006)

Designing an Evaluation• Program evaluations may be

objective-oriented

expertise-oriented

management-oriented

naturalistic/participant-oriented(McNeil, 1996).

• To serve a number of purposes, we propose a mixed model whose choice is based around four main objectives

(a) assess attainment of educational standards by students,

(b) investigate how effectively students integrate into a tertiary learning environment,

(c) determine whether students experience increases in motivation for study

(d) assist students in setting up a prospective career path

Acquisition of DataQuestion 1: What are the effects of the program on

motivation for studies, development of communication, mathematical skills and decision-making on prospective career paths?

Purpose of the Question: Assess (i) changes in student’s learning profiles (mathematical reasoning

& communication),

(ii)increases in motivation for studies and commitment towards action-planning and scheduling study activities,

(iii)integration into a university teaching and learning environment.

To Address Question 1: Data will be collected using 4 main avenues

• Academic Performance The Preparatory Program in Schools has courses with 4 assignments

each.

Based on attainments in each, marks will determine how well students are adjusting to university learning environment.

Continuous monitoring will ensure specific trends in learning attributes are recorded progressively.

Assignments have a “learning diary” and “summative essay” examine plans for future careers.

• Weekly Journal Entry As part of non-graded assessment, students will submit weekly

journals

Will provide an opportunity to self-assess their learning journey and the freedom to provide their own perspectives.

Will provide information unique to individual students (i.e. personal learning attributes).

• Covert Observations used to collect data on student’s learning profiles within a

natural learning environment

learning behaviours observed by the facilitator, allocated a rating on a numerical scale from 1-10.

frequency will be weekly

• Focus Groups pre-defined student-focus groups

stimulate discussion on a set of topics from each course.

examine study management/problem solving abilities

frequency twice a semester

Question 2: What are the effects of the program on professional development of the College staff?

Purpose of Question:(i) Obtain staffs’ perceptions on the success of the

program

(ii) Reveal challenges (e.g. resources) the institution is facing

(iii) Examine whether the program is having a positive impact on professional development

(iv) Create options for further opportunities for staff that might help improve the program delivery

To Address Question 2: Data will be collected using 2 main avenues

• ff Surveys

describe general experiences

frequency will be twice annually.

• SWOT Analysis (Staff present their roles/responsibilities) Strengths (Internal): identify new learning or pedagogical skills Weakness (Internal): identify challenges (e.g. anxiety with

“new” cohort) Opportunities (External): opportunities available (e.g. ICT

training,) Threats (External): assess the constraints (e.g. workloads)

Question 3: What extent did the program succeed in attracting community confidence?

Purpose of Question: recognise the community’s role & participation

• Parent-Teacher Interviews random batch of parents invited for interview on monthly basis. examine any growing community support for the program

• Influences Questionnaire based on influences questionnaire (Taylor & Bedford, 2004) questions centred on factors that influence a parent in

withdrawing their child or continuing 1 to 5 scale (1 = not interested, 2 = slightly, 3 = unsure, 4 = moderately, 5

= extremely interested).

Validity and Reliability• Multi-Model Approach:

• several modes of collection & merging results can lead to a valid, reliable and diverse construction of realities (e.g. Patton, 1997).

• Student Dropout: reduction in numbers can bias our results

to overcome this, we will compare characteristics of those students who have dropped out with those who remain.

if the two groups are not significantly different in terms of characteristics, then dropouts did not affect our results

(Heckman, Smith, & Taber, 1998)

• Naturalistic Methods: One example is use of covert observations

(Lynch, 1996).

• Well-defined & Systematic Procedure: can be replicated in future, thus ensuring the reliability.

Ethical Factors to Consider• Should be considered to avoid harm (Sanderson & Mazerolle, 2008).

• Ethical Clearance sought from University Ethics Committee.

• Principle of Informed Consent (Evans & Jakupec, 1996).

all participants educated about the evaluation, purpose and potential benefits (Wiles, Crow, Heath, & Charles, 2006).

• Confidentiality of Information: Using pseudonyms to avoid name disclosure (Richards & Schwartz, 2002).

• Avoiding Loss of Life-Time Opportunities: for example falling behind contemporaries, being graded as unsuccessful

or losing their career as an outcome of evaluation (Israel & Hay, 2006),

outcomes not used to process admissions in any other academic programs.

• Non-disclosure of Findings – no information disclosed to any third party

• Avoid Psychosocial Consequences: e.g. loss of self-esteem, self-concept or confidence (Bibby, 1997).

done by ensuring challenged student’s cognitive limitations are not disclosed.

• Privacy to Information: granting freedom to deny response to any particular question (Sharp,

1994).

avoid discussing characteristics that discovers the subject’s identity

• Cultural Sensitivity: exercised to protect identity of community and cultural denomination

(Hafford & Lyon, 2007).

• Data Security: deleting all subject identifiers

security devices such as passwords to protect electronic data.

Reporting Back Mechanisms(i) Progress Reports:

fortnightly reports on changes in students’ learning profiles

(ii) Newsletters: parents and school sent monthly newsletter ensuring that no specific results are disclosed to students to avoid

potential bias , “Hawthorne effect” (Kuper, Lingard, & Levinson, 2008).

(iii)Consultative Meetings: progress reports discussed in consultative meetings

(iv)General Assembly Meeting: merged in a master document Tabled in a College Assembly to gather insights and strategic

course of actions such as restructuring the program providing professional development for staff promoting the program across wider community.

Limitations of Evaluation Plan(i) Conditional Conclusions: Outcomes true under specific

conditions that are couched in probabilities rather than absolute certainty.

(ii) Time Frame: proposed 2 semester timeframe not be adequate to conduct a comprehensive assessment.

(i) Predisposition: selection based on interview so all interested students may be eager to join.

may attract those who are predisposed to a positive outcome

measuring changes in profiles may overstate the achievements

(ii) Maturation: Does not take into account effects of maturation. events outside the program cause changes in knowledge or behaviours

growth of abilities due to maturation not quantified easily

a positive outcome due to maturation treated as a plus of the program

Overcome These? Increase the length of study, 2-5 year period Use counter-control mechanisms based on previous year’s findings

References Australian Bureau of Statistics. (2006). Socioeconomic indexes for areas. Retrieved April 20, 2011, from

http://www.abs.gov.au/websitedbs/D3310114.nsf/home/Seifa_entry_page.Bibby, M. (1997). Introduction: Education research and morality. In M. Bibby (Ed.), Ethics and education research (Review of

Australian research in education no. 4). Melbourne, Vic: Australian Association for Research in Education.DETA. (2008). Next step 2008: A report on the destinations of year 12 completers from 2007 in Queensland. Queensland:

Department of Education, Training and the Arts.Evans, T., & Jakupec, V. (1996 ). Research ethics in open and distance education: Context, principles and issues. Distance

Education, 17(1), 72-94. Hafford, C., & Lyon, K. (2007). The Challenges of Conducting Evaluation in Tribal Communities. Washington, D.C: Office of Family

Assistance.Heckman, J., Smith, J., & Taber, C. (1998). Accounting for dropouts in evaluations of social experiments. Review of Economics and

Statistics, 80(1), 1-14.Hillman, K. (2005). The first year experience: The transition from secondary school to university and TAFE in Australia. Victoria,

Australia: Australian Council for Educational Research.Israel, M., & Hay, L. (2006). Research ethics for social scientists: between ethical conduct and regulatory compliance. London: Sage.Kuper, A., Lingard, L., & Levinson, W. (2008). Qualitative Research Critically appraising qualitative research. British Medical Journal,

337, a1035 Law, H., & Cook, A. (2003). Mind the gap: are students prepared for higher education?. Journal of Further and Higher Education,

27(1), 53-76. Lynch, B. (1996). Language program evaluation. Cambridge: Cambridge University Press.McNeil, J. D. (1996). Curriculum: A Comprehensive Introduction (5th ed.). Los Angeles: Harper Collins College PublishersPatton, M. Q. (1997). Utilization-focused evaluation: the new century text (3rd ed.). Thousand Oaks, California: Sage.Peel, M. (1999). Where to now? Transition from Secondary to Tertiary: A Performance Study Higher Education Series (Vol. 36, pp.

13-16). Retrieved April 20, 2011, from http://www.abs.gov.au/websitedbs/D3310114.nsf/home/Seifa_entry_page.Richards, H. M., & Schwartz, L. J. (2002). Ethics of qualitative research: are there special issues for health services research? .

Family Practice, 19(2), 35-39. Sanderson, J., & Mazerolle, P. (2008). An evaluation of the book. Retrieved April 27, 2011, from

http://www.fpq.com.au/pdf/EGB_Survey.pdf.Sharp, C. A. (1994). What is Appropriate Evaluation? Ethics and standards. Evaluation News & Comment, 3(2), 34-41. Wiles, R., Crow, G., Heath, S., & Charles, V. (2006). Anonymity and Confidentiality ESRC National Centre for Research Methods,

NCRM Working Paper Series (Vol. 2/06.).