the known unknowns: discovering what our users are doing through eesyanalytics
DESCRIPTION
TRANSCRIPT
![Page 1: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/1.jpg)
The Known Unknowns: Discovering what our Bb users are doing
through monitoring with EesyAnalytics
![Page 2: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/2.jpg)
Cardiff University
• 25,000 FTE• Russell Group & research oriented• 27 schools• 3 colleges
– Previously substantial autonomy in IT use and L&T at School level
• Our TEE platform is branded ‘Learning Central’
![Page 3: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/3.jpg)
Learning Central…
• Blackboard 9.1 SP9– Locally hosted– Course delivery system– Content management system– Community engagement system– Blackboard Mobile– Turnitin & Grademark– Learning Objects– Questionmark– Wimba Voice tools– A few custom building blocks
![Page 4: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/4.jpg)
Some (approximate) numbers
• 25,000 FTE equivalent• 6000 staff• 27 schools• 6,000 modules a year• 20,000 visits a day
![Page 5: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/5.jpg)
But what are they all doing?
• Which tools are being used?• And which are not…• Who is using them?• When are they using them?
![Page 6: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/6.jpg)
Assessing our effectiveness
• We implement tools for learning• We deliver staff development• We write support material• We visit schools & talk to staff
• How can we tell if this activity has changed anything?
![Page 7: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/7.jpg)
Our challenges
• Limited supply of resources• Plenty of demand
– For increased administrative efficiency– For improved educational quality / NSS
• Diversity of needs and practice• Always something new
– New features in Service Packs– New tools from suppliers– Emerging topics & changing practice
![Page 8: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/8.jpg)
We need more data!
• Prioritising activities– Avoid spending time on low impact areas
• Identifying areas of low/high uptake– And discovering why
• Assess whether our activity affects things– Evidence for senior managers
• Building business cases– For additional tools or resources– Evidence for deprecating obsolete tools
![Page 9: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/9.jpg)
Google Analytics
• Data on pages visited and when• Page load time• Failure to load• Dwell time• Access to individual tools
• An excellent overview, but usage is amalgamated
![Page 10: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/10.jpg)
![Page 11: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/11.jpg)
Manual reporting
• Auditing every module & assessing against benchmarks
• Extremely flexible in what it reported• Can not be done in real time; a snapshot• Expensive in staff time
Not a practical reporting solution, except as infrequent benchmarking
![Page 12: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/12.jpg)
![Page 13: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/13.jpg)
Blackboard reports
• Excellent tool for assessing the learning activity of individual users
• Data on student’s use of tests, discussion etc• Valuable tool for academics to check progress
of students
Not useful for our purposes as system admins or learning technologists
![Page 14: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/14.jpg)
BIRT
• Uses:– Bb databases– SQL– Eclipse– Building blocks
• Create custom reports run through Building Blocks
• Recently made available
Fairly technical process and runs against live database
![Page 15: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/15.jpg)
![Page 16: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/16.jpg)
Eesysoft Analytics
• Detects where user is in application– Page loads– Mouse movement over HTML elements
• Records these, and associated user role• Reports on
– What different roles have been doing– Trends in use of different tools– Comparisons between tools
![Page 17: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/17.jpg)
How it works
• Admin takes HTML snapshot of page– Define HTML elements
• JavaScript embedded in Bb hotspot• Detects what mouse pointer is over or which
page is loaded• Takes data from user profile• Sent to Eesysoft server & recorded
![Page 18: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/18.jpg)
![Page 19: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/19.jpg)
Contextual help & Eesysoft
• Our entry point to Eesysoft• Provides support to users on the basis of what
they doing at the time– Proactive pop ups for hints/issues– Via Support tab for support materials
• Makes it easier for users to get relevant support in a timely manner
• Multi-language support– Based on language pack in use
![Page 20: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/20.jpg)
Context sensitive help
![Page 21: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/21.jpg)
![Page 22: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/22.jpg)
Pop up help
• Alert users to new functions or problems• Only seen by those who it is relevant to• Only seen at the moment it is relevant to them
![Page 23: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/23.jpg)
EesyAnalytics
• Only in use a short time• Currently for SysAdmin or Learning Techs• Aiming to produce tailored reports for various
people in university• I’ve excluded some data (to improve clarity)• Graphs show %age user group
– Those who’ve accessed, not numbers
![Page 24: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/24.jpg)
What are students using in Schools
• Profile of tool usage in different Schools
![Page 25: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/25.jpg)
What are staff using in Schools…
![Page 26: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/26.jpg)
Undergraduates & postgraduates
![Page 27: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/27.jpg)
Uptake of new tools• Money well spent!
![Page 28: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/28.jpg)
Trendlines
• Pattern of tools over a year• Assessment deadline profile• Change in behaviour after training• Change in use after engagement or awareness
raising
![Page 29: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/29.jpg)
Pattern of student access to Turnitin in two Schools
![Page 30: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/30.jpg)
Language pack & other metrics
• Eesysoft can collect data from user profile• Useful for Welsh language support material• Demographic comparisons… potentially
– Do language pack users use different tools?– Does age make a difference?– Does prior education change what you do?
• Use evidence to build better courses
![Page 31: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/31.jpg)
Roles Roll Call
• Has identified a profusion of roles– 346 in total!– We have a nested structure– And a variety of ad hoc roles
• New Roles– Year of study (staff only)– Year started at University (staff & students)– Will allow us to see how use of changes
![Page 32: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/32.jpg)
What do these allow us to do?
• Focus our efforts better– Training– Support– Engagement, promotion
• Identify areas of good/bad practice– And share lessons– Help academics understand & visualise their students
use of the VLE
![Page 33: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/33.jpg)
Assessing our effectiveness
• Promotions events• Awareness raising• Staff engagement• Using roles to evidence the subsequent
actions after training
![Page 34: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/34.jpg)
Evidence for business cases
• ‘Hard’ evidence to go alongside focus groups and reported needs
• Comparing use of two tools– Use of native and externally supplied
• Highlighting use of a tool– In order to retain it– In order to demonstrate the need for support– To show when something is no longer used
![Page 35: The Known Unknowns: Discovering what our users are doing through EesyAnalytics](https://reader036.vdocuments.mx/reader036/viewer/2022070304/54c1bbf54a7959a8778b45ba/html5/thumbnails/35.jpg)
Conclusions
• Beginning of our journey• Benchmarking & strategising• Many data sources are available to us• We can build up a picture of activity in VLE• We can use that to improve our work and
demonstrate our value
• More work to be done and more possibilities to explore
Pete Obee: [email protected]