robert riordan + sprott school of business for cu …...robert riordan + sprott school of business...
TRANSCRIPT
![Page 1: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/1.jpg)
Robert Riordan + Sprott School of Businessfor CU EDC
Towards the development of a self-assessment metric for TA performance:
![Page 2: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/2.jpg)
We have 120 minutes together today Break at around 60 minutes… The logins for the machines are same as any
lab on campus – your Carleton Connect credentials
Washrooms & exits Question format
Anytime, anywhere :-)
Am I doin' good? © Robert Riordan ~ 2009 - 2011 2
![Page 3: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/3.jpg)
First me, then…
You
Department
Research interest, thesis topic
What your role as a TA involves
What feedback mechanism is in place for you… ?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 3
![Page 4: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/4.jpg)
Attend 10 workshops Write two workshop response papers (one
can be for this workshop) Write one TA Talk Article
Have you formally registered?
This applies to continuing/experienced TAs
More information at www.edc.carleton.ca
Am I doin' good? © Robert Riordan ~ 2009 - 2011 4
![Page 5: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/5.jpg)
Symbols Ouch :( Background & Performance Some caveats Information warm up – Do the Fermi Incentive Schemes Direct Data Collection Methods
Questionnaires/Surveys/Interviews Anecdotal Sample size determination Question Types
Indirect Data Collection Methods Instrumentation Types of measures
Hands-on Blogger Survey Monkey
Other Technology I’ve got feedback, now what? Summary
Am I doin' good? © Robert Riordan ~ 2009 - 2011 5
![Page 6: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/6.jpg)
There are a couple of symbols used throughout this presentation…
Participation – we talk
Hands-on – we do
Am I doin' good? © Robert Riordan ~ 2009 - 2011 6
![Page 7: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/7.jpg)
The worst teacher ever! Such a stupid and Self-righteous Prof.!!!! If my guessing was right, I think the most of the well rating about him were made by himself. How can I just describe about him!? Ugly, self-fish, rude, Shameless, and so on. Stay far far far away from him! If you could see this, please stop rating yourself anymore!
Am I doin' good? © Robert Riordan ~ 2009 - 2011 7
![Page 8: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/8.jpg)
Often, we receive only negative feedback
Get a thick skin ASAP!
How well do you take criticism?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 8
![Page 9: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/9.jpg)
Though we might not always want feedback
We all need feedback
We all want to do a good job
If we don’t get feedback, we make it up
Anyway, who cares…?
The answer is that we do care and the institution cares and for sure, your students care!
Am I doin' good? © Robert Riordan ~ 2009 - 2011 9
![Page 10: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/10.jpg)
From an instructor perspective…
It’s not always possible to give good feedback to TAs, even if one wants to, owing to:
▪ Time constraints
▪ Communication issues
▪ The sometimes subjective nature of the work we do
▪ Personality stuff – can neither give nor receive neither praise nor criticism
▪ Asynchronicity of the structure in which we work
Am I doin' good? © Robert Riordan ~ 2009 - 2011 10
![Page 11: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/11.jpg)
We will examine several strategies and tools that you can use to get an idea of how you are performing in the absence of what you consider to be satisfactory feedback from your supervisor
This stuff can be used in isolation or in combination with other feedback mechanisms to provide a more clear picture of your performance
Am I doin' good? © Robert Riordan ~ 2009 - 2011 11
![Page 12: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/12.jpg)
You should always secure permission before embarking on any interaction with your constituents
Such work might be UNPAID!
Am I doin' good? © Robert Riordan ~ 2009 - 2011 12
![Page 13: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/13.jpg)
Will a prof hang you out to dry? Let’s be realistic
▪ Teaching ratings are an important component of a prof’s own feedback loop
▪ They are important for tenure, for progression through the ranks and for employment opportunity
▪ Some might use you to deflect criticism from themselves
But let’s be honest, too…▪ It doesn’t happen very often at all…
▪ Don’t be hyper-aware of stuff like this to the point of poisoning your relationship
Am I doin' good? © Robert Riordan ~ 2009 - 2011 13
![Page 14: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/14.jpg)
Some material has been adapted from:
How to Measure Anything: Finding the Value of Intangibles in Business, 2nd Edition
Douglas W. Hubbard
John Wiley and Sons, 2010
Am I doin' good? © Robert Riordan ~ 2009 - 2011 14
![Page 15: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/15.jpg)
You tell me…
Am I doin' good? © Robert Riordan ~ 2009 - 2011 15
A quantitatively expressed reduction of uncertainty based on one or more observations
Douglas W. Hubbard
![Page 16: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/16.jpg)
Anything that can be observed can be measured
No matter how fuzzy the measurement, anything that reduces uncertainty is a measurement
So-called intangibles are measurable, too
Happiness
Satisfaction
Impact
Am I doin' good? © Robert Riordan ~ 2009 - 2011 16
Guy named Einstein: “Not everything that can be counted counts, and not everything that counts can be counted.”
![Page 17: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/17.jpg)
Enrico Fermi, Italian physicist
A father of the atomic bomb along with Robert Oppenheimer
Famously estimated the number of piano tuners in Chicago using a series of guestimations
Point is that the results of any honest measurement effort will yield a better estimate than saying “I have no idea.”
Am I doin' good? © Robert Riordan ~ 2009 - 2011 17
![Page 18: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/18.jpg)
Either individually or in groups of 2… Use what you know to estimate: # of litres of milk (chocolate + white from skim up to
homo) consumed in Ottawa in 2010
# of cigarettes smoked by youth (15-19 years old), by gender, in Ontario in 2009
# of A grades (A-, A, A+) achieved by all Carleton students in the fall term of academic year 2010
Population hints: Ottawa: ~ 900,000 (5th most populous in Canada and 2nd in Ontario)
Ontario: ~ 13,000,000 (largest in Canada and ~5th largest in NA), Canada 33,000,000
Carleton: ~ 25,000
Am I doin' good? © Robert Riordan ~ 2009 - 2011 18
![Page 19: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/19.jpg)
Milk (litres/person 2007):
2% – 38.0
1% – 18.3
3.25% – 12.0
Skim – 8.8
Chocolate – 5.7
Total ~ 83 * 900,000 = 74,520,000
Source: http://wendihiebert.com/2009/02/19/dairy-data-canadas-declining-per-capita-milk-consumption/
Am I doin' good? © Robert Riordan ~ 2009 - 2011 19
![Page 20: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/20.jpg)
Cigarettes smoked by youths (2008 data) 14.8% were current smokers
▪ 8.6% were daily and 6.2% non-daily smokers
Daily smokers averaged 12.2 cigarettes/day▪ Assume non-daily smokers smoke at 1/3 of daily smokers’ rate = 4
cigarettes/day
16.8% of males and 12.7% of females were current smokers Population aged 15-19 on July 1, 2009
▪ Male: 1,153,334 ~ Female: 1,098,791
Total: ~> 1,000,000,000 (Canada); 420,000,000 (Ontario)
http://www.tobaccoreport.ca/ytu.htmlhttp://www.statcan.gc.ca/daily-quotidien/091127/t091127b2-eng.htmhttp://www.statcan.gc.ca/daily-quotidien/090326/t090326a2-eng.htm
Am I doin' good? © Robert Riordan ~ 2009 - 2011 20
![Page 21: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/21.jpg)
Grades…
Am I doin' good? © Robert Riordan ~ 2009 - 2011 21
0
5
10
15
20
25
F D C B A
![Page 22: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/22.jpg)
Once you have permission, determine if: Prof will allow bonus marks for students to
participate in feedback What are the issues with incentive? Make sure you have a mechanism to track unique
student evals – duplicates are bad data
Self-selection bias (non-random sample):▪ Students who participate will likely fall into two groups:
▪ High achievers (need good grades for their own feedback loop )
▪ Low achievers (desperate to pass in any way possible)
▪ The great mass in the middle is missing
Am I doin' good? © Robert Riordan ~ 2009 - 2011 22
![Page 23: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/23.jpg)
These measurements require direct contact with the student in order to gather data
They require explicit permission from the instructor or department Make sure there are no privacy violations
Strongly recommend that you get approval of the instrument before administering it
Direct contact can be either anonymous or identified There are issues with each…
Am I doin' good? © Robert Riordan ~ 2009 - 2011 23
![Page 24: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/24.jpg)
Email How to harvest emails?
Web Course website
WebCT
Your own site
Blog Online surveys
PHYSICAL DISTRIBUTION
In class or in lab or tutorial session or office hours?
Inserted in tests/exams? Included in notes/lab
material? Others?
ELECTRONIC DISTRIBUTION
Am I doin' good? © Robert Riordan ~ 2009 - 2011 24
![Page 25: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/25.jpg)
PHYSICAL COLLECTION
Not anonymous but almost… especially in large class settings or with drop boxes Drop box at department
office?▪ With permission
Drop box at your office?▪ Security?
Submit along with test/assignment/lab?
Others?
ELECTRONIC COLLECTION
If not part of the tool (as in an online survey)
Can be anonymous Returned as body of or as
attachment to email
Anonymous issues Can’t control:
Multiple submissions No submission Quality of responses
Am I doin' good? © Robert Riordan ~ 2009 - 2011 25
![Page 26: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/26.jpg)
Create an instrument from which it is easy to extract data:
Numeric/multiple choice/Likert scale questions…
Fewer items the better
Decide on a focal element or two and stick to them!
Example foci:
▪ Turnaround speed
▪ Richness of feedback
▪ Availability
▪ Others…
Am I doin' good? © Robert Riordan ~ 2009 - 2011 26
![Page 27: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/27.jpg)
When to interview Office hours
Test/assignment/lab reviews
Before or after class or during a break
Scheduled time▪ By invitation (you invite at specific time) or by appointment
(their initiative)
Environment is important▪ Confidentiality is important
▪ Comfort is important
▪ Beware the interviewer effect – do you control their marks?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 27
![Page 28: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/28.jpg)
Simply ask students about their experience: Satisfaction with your service
▪ Be sure to separate you from the rest of the infrastructure
▪ This will be important no matter which direct measure you use▪ Students sometimes don’t differentiate you from the prof from the lectures
from the lab machines from the textbook…
Difficult to get an empirical measure from this technique▪ Can’t compare across interviewers as it’s very free-form.
Can suggest topics for a more focussed instrument (in the Bayesian sense)
Again, be aware that people often use an opportunity like this to vent… you are an easy and available target
Am I doin' good? © Robert Riordan ~ 2009 - 2011 28
![Page 29: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/29.jpg)
Sample size
This can be simple or sophisticated… let’s be simple about it:
▪ Let’s assume that you have an item such as the following: I am satisfied with the service provided by the TAs in this class. Agree
or Disagree?
▪ What does it mean if 62% agreed? Or if 29% disagreed?
▪ How many people do I need to ask?
▪ What’s the margin of error?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 29
![Page 30: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/30.jpg)
Let’s start at the obvious extremes in terms of sample size If your sample was ONE and that person disagreed, you’d
have a dismal satisfaction rate of 0% OR a dissatisfaction rate of 100%. Time to find the O-Train trestle?
Obviously not…
Facts: Required sample size is a function of three simple things: Confidence level Confidence interval Population size
Am I doin' good? © Robert Riordan ~ 2009 - 2011 30
![Page 31: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/31.jpg)
Confidence level
Do you want to be 95% sure? Or 99% sure?
Sure of what?
Confidence interval
Sure that your observation is +/- a CI of the true population figure (the real, underlying answer to the question that would be exposed if you interviewed everyone in the population of interest)
Population size
From how many are we sampling?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 31
![Page 32: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/32.jpg)
Example with lexicon: If 68% agreed with your statement and you want a:
▪ Confidence level of 95% and▪ a confidence interval of 5▪ from a population of 200▪ You would need 132 observations (persons asked) to be able to
state that:I am 95% sure that between 63% and 73% of students in this class agree
that they are getting good service…
Stated another way:▪ If you did 100 surveys of 132 students from this population of 200,
95% of the time the observed percentage agreeing would be between 63 and 73. Five times it would be outside this window, even if the true population percentage is 68%.
Am I doin' good? © Robert Riordan ~ 2009 - 2011 32
![Page 33: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/33.jpg)
Let’s go to a website and fiddle around… http://www.surveysystem.com/sscalc.htm
Level Interval Population Sample Size?
95 5 60 52
95 5 300 169
95 1 60 60
99 5 300 207
95 5 30,000,000 384
99 1 30,000,000 16,632
Am I doin' good? © Robert Riordan ~ 2009 - 2011 33
![Page 34: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/34.jpg)
Sometimes we don’t have the luxury of being able to specify sample size
So we can work backwards Given the numbers we have, how do we
estimate confidence in the measurement?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 34
![Page 35: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/35.jpg)
This table shows the 90% CI for small samples (Lower Bound and Upper Bound shown as %) Simply look up the sample size in the column and the row with the number of hits. To estimate more than 10 hits, invert the table and estimate ‘misses’ …
1 2 3 4 6 8 10 15 20 30
0 2.5-78 1.7-63 1.3-53 01.0-45 0.7-35 0.6-28.3 0.5-23.9 0.3-17.1 0.2-13.3 0.2-9.2
1 22.4-97.5 13.5-87 9.8-75.2 07.6-65.8 05.3-52.1 4.1-42.9 3.3-36.5 2.3-26.4 1.7-20.7 1.2-14.4
2 36.8-98.3 25-90.3 18.9-81 12.9-65.9 9.8-55 07.9-47.0 5.3-34.4 4.0-27.1 2.7-18.9
3 47-98.7 34.3-92.4 22.5-78 16.9-66 13.5-57 9.0-42 6.8-33 4.5-23
4 55-99.0 34.1-87 25.1-75 20-65 13-48 9.9-38 6.6-27
5 48-94.7 34.5-83 27-73 17.8-55 13.2-44 8.8-31
6 65-99.3 45-90 35-80 22.7-61 16.8-49 11.1-35
7 57-95.9 44-87 28-67 21-54 14-38
8 72-99.5 53-92 33-72 25-58 16-42
9 64-96.7 39-77 29-63 19-45
10 76-99.6 45-82 33-67 21-49
Sample Size
Nu
mb
er o
f “h
its”
in S
amp
le
...
Am I doin' good? © Robert Riordan ~ 2009 - 2011 35
![Page 36: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/36.jpg)
Two basic genres of question: Quantitative (hard metric)
▪ What was your age at your last birthday?
Qualitative (open-ended)▪ How do you feel about getting older?
There are pros and cons to each… Qualitative is more time consuming to analyse
and is difficult to summarise
Quantitative is easier to work with but not nearly as rich… nothing is free.
Am I doin' good? © Robert Riordan ~ 2009 - 2011 36
![Page 37: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/37.jpg)
When you can’t get direct access to students either personally or anonymously…
Indirect measures are sometimes called proxyor surrogate measures because they stand infor direct measures…
Will need approval (and active participation from your department as well) to get some of these data…
Some can be had for the price of a coffee :-)
Am I doin' good? © Robert Riordan ~ 2009 - 2011 37
![Page 38: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/38.jpg)
Compare anecdotally with other TAs Across sections of the same course in the same term Across sections of the same course in different terms
(if possible – see below for a solution going forward) Across different courses with similar structures in the
same or different terms Start a process/data repository to keep
measurements for future reference Great longitudinal data source – can be used for
several purposes (benchmarking, marketing, etc…) Maybe your department would be interested in
supporting such an enterprise?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 38
![Page 39: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/39.jpg)
Instrumentation
Either alone or in a group, you can develop measures to give proxy indicators of your efficiency and effectiveness
Speed:
▪ Minutes per task (per test, per paper, per experiment)
Accuracy:
▪ Number of returns, complaints, appeals, etc
If you use speed, you really must also use accuracy
Speed is a relative term… faster is not always better!
Am I doin' good? © Robert Riordan ~ 2009 - 2011 39
![Page 40: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/40.jpg)
Instrumentation (cont) If you are comparing across sections or years or
instructors, you really must control for (take into account) certain potential and implied differences…
These include:▪ Instructors▪ Curriculum▪ Time of day (huge effect on class demographics and
satisfaction!)▪ Term (summer is way different!)▪ TAs▪ Environment (room, lab, etc.)▪ Others?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 40
![Page 41: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/41.jpg)
What kind of measures can you create? Attendance rates for lectures, labs, tutorials,
office hours▪ Need total numbers for denominator
▪ Can average them over a term to remove peaks caused by approach of tests/exams/assignments
▪ Can remove the peak days and look at ambient levels
▪ You are looking for patterns and trends here
▪ This type of data is difficult to interpret in isolation and must be considered in terms of the contaminants mentioned on the previous slide
Am I doin' good? © Robert Riordan ~ 2009 - 2011 41
![Page 42: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/42.jpg)
Measures (cont) Rates of contact by type of contact:
▪ Argue about mark
▪ Request for remark
▪ Clarification
Various denominators are possible, viz:▪ Total students in class, lab, etc
▪ Total students for whom you are responsible
▪ Total visits/contacts
▪ Others?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 42
![Page 43: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/43.jpg)
Measurements (cont)
Average and maximum discrepancy in grade adjustment, by various denominators, viz:
▪ Total students
▪ Total adjustments made
▪ Others?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 43
![Page 44: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/44.jpg)
Secondary or administrative data With the permission of your department, you
can use aggregate data such as: DFW (Drop, Fail, Withdraw) rates
Grade distributions over time (this is very sensitive data…) in as fine a detail as you can get them, viz:▪ By term, instructor, enrolment, etc.
▪ By numeric (if available) and/or letter grade
▪ By time of day
▪ Any other detail available
Am I doin' good? © Robert Riordan ~ 2009 - 2011 44
![Page 45: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/45.jpg)
Blogging A blog is a web log … a recording of random and/or
purposeful messages to the world hosted on a blogging site on the web
Blogs have interesting features for us▪ They are free
▪ They allow anonymous feedback
▪ They allow very simple surveys
▪ They can provide links to documents (if you have a place to host the docs)
▪ They can provide links to other, more sophisticated (but still free) online survey tools
Am I doin' good? © Robert Riordan ~ 2009 - 2011 45
![Page 46: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/46.jpg)
Let’s go and take a BLOG survey now… Fire up your machine and navigate to: http://carletonta.blogspot.com
Don’t click thesurvey link yet!
Am I doin' good? © Robert Riordan ~ 2009 - 2011 46
![Page 47: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/47.jpg)
Now let’s create a BLOG at:
http://www.blogger.com
Am I doin' good? © Robert Riordan ~ 2009 - 2011 47
![Page 48: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/48.jpg)
If you have a Google (Gmail) account, log in If not, click on
Blogger requires a Google account
If you don’t have one, take a moment to create one if you want one…
From here, just follow the directions to create a blog…
I’m here for moral support :-)
Am I doin' good? © Robert Riordan ~ 2009 - 2011 48
![Page 49: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/49.jpg)
Now let’s go back to the BLOGGER survey at: http://carletonta.blogspot.com
NOW click thesurvey link!
Am I doin' good? © Robert Riordan ~ 2009 - 2011 49
![Page 50: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/50.jpg)
Now that we have some data in there, let’s take a quick look at the output
Login
My Surveys
Analyze
Am I doin' good? © Robert Riordan ~ 2009 - 2011 50
![Page 51: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/51.jpg)
Certain functions are not available on the free basic account
Am I doin' good? © Robert Riordan ~ 2009 - 2011 51
![Page 52: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/52.jpg)
IF we have time, we’ll create our own survey on http://surveymonkey.com …
We can mix and match questions and question types…
Experiment with scales
Am I doin' good? © Robert Riordan ~ 2009 - 2011 52
![Page 53: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/53.jpg)
Am I doin' good? © Robert Riordan ~ 2009 - 2011 53
Source: http://www.tagcrowd.com/
![Page 54: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/54.jpg)
WebCT does not seem to have any tools that we can use for our purposes, other than being a vessel for providing the download of a physical survey or a link to a blog/survey…
Twitter…
Could really get out there with this technology…
Would need to designate some Tweeters in the class (perhaps incentive-based) and follow them?
Any other suggestions on how to use it?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 54
![Page 55: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/55.jpg)
Facebook?
Create a group and log feedback
YouTube?
Broadcast announcements…
▪ How to get feedback?
Am I doin' good? © Robert Riordan ~ 2009 - 2011 55
![Page 56: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/56.jpg)
How to interpret?
That depends…
Teaching Portfolios
Academic appointments frequently require a teaching portfolio
▪ Evidence of track record and excellence in teaching including concrete EVALUATION DATA
▪ This is a great start…
Am I doin' good? © Robert Riordan ~ 2009 - 2011 56
![Page 57: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/57.jpg)
Sometimes and for a variety of reasons, we don’t get the feedback that we need
If, however, we are creative, inventive, diligent and motivated, we can find and do all kinds of things in order to satisfy our innate need for feedback…
There are basically two types of data:
Direct and indirect
And responses can be anonymous or not…
Am I doin' good? © Robert Riordan ~ 2009 - 2011 57
![Page 58: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/58.jpg)
You can cooperate with others, or go it alone You can use paper or electronic means You can use secondary (administrative) data
if you cannot get direct (primary) data You can start a programme of data collection
and archive in order to support the initiative of improving feedback for TAs and thus positively impacting the quality of education at Carleton… a laudable goal indeed…
Am I doin' good? © Robert Riordan ~ 2009 - 2011 58
![Page 59: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/59.jpg)
It’s been a great pleasure :) My contact information:
Rob RiordanSprott School of [email protected]
Don’t be strangers! If you’d like a copy of the presentation, email
me :-)
Am I doin' good? © Robert Riordan ~ 2009 - 2011 59
![Page 60: Robert Riordan + Sprott School of Business for CU …...Robert Riordan + Sprott School of Business for CU EDC Towards the development of a self-assessment metric for TA performance:](https://reader033.vdocuments.mx/reader033/viewer/2022052518/5f0855507e708231d4217da4/html5/thumbnails/60.jpg)
Am I doin' good? © Robert Riordan ~ 2009 - 2011 60