aura programme: collect learning assessment data

Engaged Excellence in Research & Teaching Practicesaura

Collect Learning Assessment Data

Teaching Course 1, Day One, Session 4

Mood Monitor

Session’s Learning OutcomesBy the end of this session you will:

1. Recognise why data collection is important

2. Explain that data collection can:

1. Take place at different times (when);

2. Be used in different ways (how);

3. Measure different things (what)

3. Identify best practices in designing data collection questions

When is Formative Assessment Carried out?A. Before an intervention

B. During an intervention

C. Before and during an intervention

D. After an intervention

How do you feel about the prospect of a student evaluating your classes?A. It is the academics job to assess the student, not the other way

around

B. I collect student feedback only because I am asked to.

C. I incorporate student feedback into my teaching strategies and

curriculum design

D. I am unsure

What is your average class size?

A. 0-25

B. 26-50

C. 51-75

D. 75-100

E. 100 Plus

Group Activity

What forms of learning assessment data do you collect?

Discuss in groups and feed into the class

(5 minutes)

When, How, What?The 3 questions are interrelated

When: before training, after training or several months/years after

What: attainment; satisfaction rates, attitudes and long term behaviour and impact

How: qualitative or quantitative

When are you Measuring?

Formative assessment Summative assessment

Pre-training assessment

During training Post-training

How are you Measuring?

Pavel Ševela / Wikimedia Commons Nicholas/Wikimedia Commons

Qualitative Quantitative

What are you Measuring?Before Training After Training Long Term

Applicant suitability Satisfaction with course content

Change in skills/attitudes (was it retained?)

Participant profile Satisfaction with course delivery

Change in behaviour

Participant’s prior experience

Change in skills Impact

Expectations Change in attitudes ‘Externalities’

Group Activity 2: Design a SurveyPlease get into groups and design 5 questions that you would like to ask your students – use the following table:

Please try to get a mix of the When; What; and How

Question When? What? How?E.g. What are you hoping to gain from this course? Pre-course Background Qualitative

E.g. What industry are you working in now?

Post-course Career Quantitative

Group Activity 3: Critique a Survey1. Please get into groups and take a look at the

survey handout (5 mins)

2. Go over the survey and make amendments to existing questions where you feel you can improve them (10 mins)

Good Practices in Setting Data Collection Questions

Avoid Vanity Metrics

Vanity metrics include questions that participants feel compelled to

give a positive answer to

E.g. Just how satisfied were you with the our course?

• Often participants tell you what they think you want to hear

• This question is not explicit about the quality of the course

• Be specific and well directed in your questions to avoid vanity metrics

Use Likert ScalesA Likert Scale to score responses:1. Use 1-5 or 1-10

• Note: research shows individuals tend to choose the median score (e.g. 3 in 1 to 5 scale)

• Always indicate which is the high and low value (they are usually in ascending order but be explicit about this)

2. Use balanced and meaningful supporting text:• Strongly disagree• Disagree• No opinion• Agree• Strongly Agree

Likert Scales: Video

https://www.youtube.com/watch?v=q3FnFTJ528s

Use Exhaustive & Plausible QuestionsWhen supplying multiple-choice options ensure:

• Plausibility (more applicable to tests)

• Cater for all responses:• By ensuring that the categories are comprehensive• Allow for a free response option (i.e. ‘other’)

Plausibility: Video

https://www.youtube.com/watch?v=NXvmkTC9rcA

Collect Demographic Data

Collect profile data about the contributor, such as:• Gender: • Age: • Preferred contact:

• Email • Telephone / Mobile

• Organisation: • Job Title:

Demographic Data: Video

https://www.youtube.com/watch?v=hdRaMg3hBtk

Good Practice List1. Be specific – defining any technical words2. Avoid leading questions3. Avoid any ambiguity4. Avoid vanity metrics5. Be consistent in your questions6. Be careful with your sequence of questions7. Strike a balance between qualitative and quantitative8. Avoid asking too many questions (this is context specific)9. Avoid asking too few questions (also, context specific)10. Be creative

This work is licensed under a Creative Commons Attribution-Non-commercial ShareAlike 3.0

The content is authored by:Jagdeep ShokarMonitoring and Evaluation Advisor,Institute of Development Studies, j.shokar@ids.ac.uk