Developing and Assessing the Developing and Assessing the Research Skills of Students in Research Skills of Students in

Engineering CoursesEngineering Courses  

Dr Said Al-SarawiDr Said Al-SarawiResearch Skill Development and Assessment Research Skill Development and Assessment

ALTC Project MemberALTC Project Member

School of Electrical and Electronic EngineeringSchool of Electrical and Electronic Engineering

University of AdelaideUniversity of Adelaide

A seminar for Curtin Engineering FacultyA seminar for Curtin Engineering Faculty

OutlineOutline MotivationMotivation Students and staff issuesStudents and staff issues RSD Framework in AustraliaRSD Framework in Australia Benefits of using RSDFBenefits of using RSDF What is RSD Framework?What is RSD Framework? Case Studies – From 1Case Studies – From 1stst Year to Master Level Year to Master Level Other Dimensions of RSD FrameworkOther Dimensions of RSD Framework RSD at the Program Level !RSD at the Program Level ! RSD in conclusionRSD in conclusion Discussion and future workDiscussion and future work

Motivation (1/2)Motivation (1/2) PhD completion PhD completion rate were doubled for students who had rate were doubled for students who had

participated in undergraduate research (Bauer & participated in undergraduate research (Bauer & Bennett, 2003)Bennett, 2003)

Performance based Performance based research funding research funding (old RQF, ERA)(old RQF, ERA) Performance based learning and teaching support – Performance based learning and teaching support –

Learning and Teaching Performance Fund Learning and Teaching Performance Fund (DEST, 2006)(DEST, 2006) Undergraduate research has been associated with Undergraduate research has been associated with

higher level of student satisfaction and higher level of student satisfaction and generic skills generic skills developmentdevelopment (Kardash, 2000) (Kardash, 2000)

Factors affecting skills implementation are (Lucas et al, Factors affecting skills implementation are (Lucas et al, 2000)2000)

ScepticismScepticism of the message, the messenger and its vocabulary of the message, the messenger and its vocabulary The skills demanded The skills demanded lack claritylack clarity, consistency and recognisable , consistency and recognisable

theoretical basetheoretical base The skills are dependent on The skills are dependent on discipline areadiscipline area

Motivation: Student and Staff issuesMotivation: Student and Staff issues (2/2) (2/2)

StaffStaff StudentStudent

How can I How can I developdevelop and and assessassess students’ research students’ research skills?skills?

What is meant by What is meant by Research and what skills Research and what skills are needed?are needed?

Is there a systematic, Is there a systematic, explicit approach that can explicit approach that can be used?be used?

How I can acquire and How I can acquire and develop these skills? develop these skills?

Is this issue only limited to Is this issue only limited to the engineering discipline?the engineering discipline?

How are they relevant to How are they relevant to my career aspirations? my career aspirations?

How can I remove the How can I remove the “subjectiveness” from the “subjectiveness” from the assessment?assessment?

How will my skills be How will my skills be assessed?assessed?

RSDF in AustraliaRSDF in Australia

• The University of Adelaide (John Willison)− Human Biology (Eleanor Peirce & Mario Ricci)

− Electrical Engineering Masters by Coursework (Said Al-Sarawi and

Brian Ng)

− Clinical Nursing (Frank Donnelly)

− Petroleum Engineering (Steve Begg)

− Introductory Academic Program (Richard Warner)

− English (Joy McEntee)

− Dentistry (Vicki Skinner and Leonard Crocombe)

− Oral Health (Sophie Karanicolas and Cathy Snelling)

− Software Engineering (Li Jiang)

− Veterinary Science (Susan Hazel)

RSDF in Australia (RSDF in Australia (Conts.Conts.))

• Macquarie University (Psychology: Judi Homewood)

• Monash University (Business Ethics: Jan Schapper;

Sue Mayson: Business; Glen Croy: Tourism)

• University of Melbourne (Business Law: Eu-Jin Teo)

• University of South Australia (Introduction to Tertiary

Learning, 2008: Rowena Harper)

The facets of student researchThe facets of student researchIn researching, students:In researching, students:

1.1. embark on an inquiry embark on an inquiry and so determine a need for and so determine a need for knowledge/understandingknowledge/understanding

2.2. find/generatefind/generate needed information using appropriate needed information using appropriate methodologymethodology

3.3. critically evaluate critically evaluate information/data and the process to information/data and the process to find/generatefind/generate

4.4. organiseorganise information collected/generated information collected/generated5.5. synthesisesynthesise and analyse and apply new knowledge and analyse and apply new knowledge6.6. communicatecommunicate knowledge and the processes used to generate knowledge and the processes used to generate

it, with an awareness of ethical, social and cultural issues.it, with an awareness of ethical, social and cultural issues.

(Willison & O’Regan, 2007)(Willison & O’Regan, 2007)

Level 1Level 1

Facet A) Embark on Inquiry Facet A) Embark on Inquiry Respond to questions / tasks arising explicitly from Respond to questions / tasks arising explicitly from a closed inquiry.a closed inquiry.

Facet E) Synthesis, analysis, application Facet E) Synthesis, analysis, application Ask Ask questions of clarification / curiosity.questions of clarification / curiosity.

Learning to Frame Research Learning to Frame Research QuestionsQuestions

Level 2Level 2

Facet A) Embark on Inquiry Facet A) Embark on Inquiry Respond to questions / tasks required by and implicit Respond to questions / tasks required by and implicit in a closed inquiry.in a closed inquiry.

Facet E) Synthesis, analysis, application AFacet E) Synthesis, analysis, application Ask relevant, sk relevant, researchable questions.researchable questions.

Learning to Frame Research Learning to Frame Research QuestionsQuestions

Level 3Level 3

Facet A) Embark on Inquiry Facet A) Embark on Inquiry Respond to questions / tasks generated from a closed Respond to questions / tasks generated from a closed inquiry.inquiry.

Facet E) Synthesis, analysis, applicationFacet E) Synthesis, analysis, applicationAsk rigorous, researchable questions based on new Ask rigorous, researchable questions based on new understandings.understandings.

Learning to Frame Research Learning to Frame Research QuestionsQuestions

Why use RSD approaches? (1/2)Why use RSD approaches? (1/2) Benefits of using the RSD for students Benefits of using the RSD for students

(according to Eleanor Peirce and Mario Ricci, (according to Eleanor Peirce and Mario Ricci, Medical Sciences, Uni of Adelaide)Medical Sciences, Uni of Adelaide)

Their research skills in our course have Their research skills in our course have improved. improved.

They understand much more clearly what is They understand much more clearly what is expected of them. expected of them.

They know exactly where they need to develop, They know exactly where they need to develop, thanks to feedback. thanks to feedback.

Benefits for Lecturers (2/2)Benefits for Lecturers (2/2)

We can give feedback on assessment tasks more We can give feedback on assessment tasks more accurately and efficientlyaccurately and efficiently; we can give the same ; we can give the same quality feedback with less writing, and faster. quality feedback with less writing, and faster. We can get a much better idea of where our We can get a much better idea of where our students are from a students are from a quick analysisquick analysis of the RSD of the RSD results than from a detailed analysis of standard results than from a detailed analysis of standard marks. marks. We can easily We can easily match assessment match assessment tasks with tasks with course objectives, and course objectives with the course objectives, and course objectives with the University’s Graduate Attributes. University’s Graduate Attributes.

(according to Eleanor Peirce and Mario Ricci, Medical Sciences, Uni of Adelaide)

Case Study 1: Human Biology - 1Case Study 1: Human Biology - 1stst Year Year

O Week RSD

I II

A

F

Read the two short then complete tasks 1 and 2.

Task 1: Integrate the information presented in the two articles to write your own dot-point noteson the worksheet attached. To do this:

• Identify 3-4 key ideas from the articles• Use these key ideas to formulate

headings and underline each• Make bullet-point notes and list them

under these headings.• After each point, indicate its source, i.e.

whether the idea came from article 1, article 2, or both

• Provide a title that embodies the content of your notes.

Task2:Which of the two articles do you consider to be the better source? On what characteristics/features of the article have you based your choice?

Case Study 1: Human Biology - 1Case Study 1: Human Biology - 1stst Year Year

Literature Research Skill Stream

Laboratory Research Skill Stream

Field and literature researchSemester 1 Semester 2

Case Study 1: Human Biology - 1Case Study 1: Human Biology - 1stst Year Year

O Week RSD

I II

A

F

Lit Week RSD 1

I II III

A

F

Lit Week RSD 1

I II III

A

F

Small Group InquiryI II III IV

A

F

Lab RSD 1

I II

A

F

Lab RSD 2

I II

A

F

Individual Open Inquiry

I II III IV

A

F

Novelty in the engineering caseNovelty in the engineering case

The differentiation points:The differentiation points: Clearly state the need to research skill Clearly state the need to research skill

developmentdevelopment The students are required to identify a The students are required to identify a

gap in knowledge for each of the gap in knowledge for each of the chosen topics (the unknown)chosen topics (the unknown)

Rigorous literature researchRigorous literature research

Case study 2: Photonic and Case study 2: Photonic and Communication – Master CourseCommunication – Master Course

Masters (coursework) course at EEEMasters (coursework) course at EEE Course run by senior colleague in EEECourse run by senior colleague in EEE 2 units of lectures/exams2 units of lectures/exams 1 unit of 1 unit of literature research projectliterature research project Students seek supervisors individuallyStudents seek supervisors individually Topic chosen by student, but requires approval by supervisorTopic chosen by student, but requires approval by supervisor

DemographicsDemographics Almost all are international students; usually East Asian backgroundAlmost all are international students; usually East Asian background Technically capable, but generally lack prior experience in Technically capable, but generally lack prior experience in

conducting researchconducting research Goal: produce a high quality review paper on chosen topicGoal: produce a high quality review paper on chosen topic Mostly negative experiences in 2005Mostly negative experiences in 2005

Cohort lacked basic literary research skillsCohort lacked basic literary research skills Unstructured approach towards projectUnstructured approach towards project Low quality final review papersLow quality final review papers

Process DetailsProcess Details1.1. Initial diagnostic taskInitial diagnostic task

• Supervisor supplies Supervisor supplies twotwo technical papers technical papersDifferent levels (magazine, journal) for contrastDifferent levels (magazine, journal) for contrast

• Students summarise and compare the Students summarise and compare the keykey points from both sources points from both sources in one in one structured, bulleted liststructured, bulleted list

• Detailed supervisor feedback in written formDetailed supervisor feedback in written form2.2. Knowledge accumulation phaseKnowledge accumulation phase

• More articles added to the reading listMore articles added to the reading listContinually add to an organic structured, bulleted listContinually add to an organic structured, bulleted list

• Strategy on further readingStrategy on further readingStudent applies critical evaluation on suitability of sources with Student applies critical evaluation on suitability of sources with

supervisor inputsupervisor input• Fortnightly workshops for group presentations & discussionsFortnightly workshops for group presentations & discussions

Supervisor supplies feedback; optional: external advice (CLPD)Supervisor supplies feedback; optional: external advice (CLPD)3.3. Writing phaseWriting phase

• Student writes review paper based on listStudent writes review paper based on list• Supervisor feedback on first draft approx a week before submissionSupervisor feedback on first draft approx a week before submission

Outcome and EvaluationOutcome and Evaluation Outcomes from 2006Outcomes from 2006

Total of 6 students plus1 external (industry) student as controlTotal of 6 students plus1 external (industry) student as control Qualitatively, much improved papers compared to 2005Qualitatively, much improved papers compared to 2005

• Coherent structures, logical arguments, conciseness, respect of Coherent structures, logical arguments, conciseness, respect of referencing practicereferencing practice

Student perspectiveStudent perspective• struggled to cope with demands of research among the pressures struggled to cope with demands of research among the pressures

of regular courseworkof regular coursework• English as second language remained a great barrierEnglish as second language remained a great barrier

Framework “matrix” useful for quantitative assessmentFramework “matrix” useful for quantitative assessment Unexpected benefit – reduction in plagiarismUnexpected benefit – reduction in plagiarism

Turn-it-in software reported major improvements from 2005 Turn-it-in software reported major improvements from 2005 Reports tend to be on conservative side.Reports tend to be on conservative side.

Case 3: Final Year Project (in Progress)Case 3: Final Year Project (in Progress)

The set of project deliverables are:Semester 1 • Formative literature assessment

(diagnostic value?)• Proposal seminar• design document/interim report• Critical design review (peer review)

Semester 2 • final report• final seminar• project log books/wiki/blog/email log

… etc

“We have been concerned about a degree of disconnect between the desired leaning outcomes and the assessment methods used in these projects.” HoS of School of EEE, The University of Adelaide.

RSD: Useful First-year to PhD?RSD: Useful First-year to PhD?

Variety of RSD approachesVariety of RSD approachesFive distinct approaches have emerged in the use of the RSD so far: •Rubrics base to assess the profile of skills for each student, as demonstrated by Human Biology rubrics.  this is the most common approach, and is used by most disciplines.•A lock-step approach, whereby students are kept 'in formation' and progressively and corporately develop one level of skills at a time.  This is shown by the Nursing RSD rubrics. •Grading with a specific grading, by incorporation into SOLO taxonomy to define grading within a specific level set, as used by Dentistry.•Skill Evaluation, to evaluate the skills and levels required by existing assessments. •Identification of research skills of higher degree students, Masters by Research and PhD students and/or their supervisors/advisors to locate their present skill set and plot future directions and development needs. 

(SOLO: Structured Observed Leaning Outcomes)

• Degree of Autonomy

• Degree of Academic Rigour

• Degree of Conceptual Demand

• Degree of disciplinary knowledge required

• Status of knowledge being pursued

Other Dimensions of RSD Other Dimensions of RSD

RSD Framework at Program Level !RSD Framework at Program Level !No studies that consider student outcomes of the explicit development and assessment of research skills over a

whole undergraduate or masters-by-coursework program

Undergraduate Level• Bachelor Oral Health, Adelaide Uni, AU

• Bachelor of Media Studies , Adelaide Uni, AU• Bachelor of Science, Adelaide Uni, AU• Bachelor of Nursing, Adelaide Uni, AU• Bachelor of Business, Monash Uni, AU• Bachelor of Science (Psychology), Macquarie University, AUPostgraduate Level• Master of Engineering (Advanced) (Electrical), Adelaide Uni, AU• Bridging Program for International PhD students, Adelaide Uni, AU• Masters and PhD (Nursing), PhD (Nursing)• Trinity College Dublin, Ireland• Master of Business, Monash University, AU

2626

Provides the Big Picture and relates this to the Provides the Big Picture and relates this to the assessment details for course coordinators, assessment details for course coordinators, lecturers, tutors, and especially studentslecturers, tutors, and especially students

Informs assessment-first curriculum redesignInforms assessment-first curriculum redesign Same ‘facets’ for multiple assessments, various Same ‘facets’ for multiple assessments, various

levelslevels Explicit &Transparent assessment criteriaExplicit &Transparent assessment criteria Coherent & Incremental skill developmentCoherent & Incremental skill development Revisited & (potentially) Cyclic Conceptual Revisited & (potentially) Cyclic Conceptual

FrameworkFramework

RSD in conclusionRSD in conclusion

Discussion and Future workDiscussion and Future work How to integrate the framework into other How to integrate the framework into other

Programs?Programs? How will the implementation of RSD framework How will the implementation of RSD framework

affect academic workload?affect academic workload? How to monitor and assess student’s progress?How to monitor and assess student’s progress? How can this be implemented for larger class How can this be implemented for larger class

sizes?sizes? How can the framework adopted for non-literary How can the framework adopted for non-literary

research skills?research skills?

ReferencesReferences

This session was funded by an Australian Learning and Teaching Council Grant

http://www.adelaide.edu.au/clpd/rsd

1.1. Bennett, N., Dune, E. & Carré, C. (2000). Skills Development in Higher Education and Bennett, N., Dune, E. & Carré, C. (2000). Skills Development in Higher Education and Employment. (Society for Research into Higher Education & Open University Press).Employment. (Society for Research into Higher Education & Open University Press).

2.2. Stevens, C. & Fallows, S.J. (2000). Integrating Key Skills in Higher Education: Stevens, C. & Fallows, S.J. (2000). Integrating Key Skills in Higher Education: employability, transferable skills and learning for life. Routledge, ISBN 0749432659.employability, transferable skills and learning for life. Routledge, ISBN 0749432659.

3.3. Lucas, U., Cox, P., Croudace, C. and Milford, P. (2004). Lucas, U., Cox, P., Croudace, C. and Milford, P. (2004). “Who Writes This Stuff?”: “Who Writes This Stuff?”: Students’ Perceptions of Their Skills Development. Students’ Perceptions of Their Skills Development. Teaching in Higher EducationTeaching in Higher Education, 9(1), , 9(1), 55-68.55-68.

4.4. Willison, J.W. & O’Regan, K. (2007). Commonly known, commonly not known, totally Willison, J.W. & O’Regan, K. (2007). Commonly known, commonly not known, totally unknown: A framework for students becoming researchers. unknown: A framework for students becoming researchers. Higher Education Research Higher Education Research and Development,and Development, 26(3) 26(3)..

RSD Web SiteRSD Web Site

AcknowledgementAcknowledgement