The Known Unknowns: Discovering what our Bb users are doing
through monitoring with EesyAnalytics
Cardiff University
• 25,000 FTE• Russell Group & research oriented• 27 schools• 3 colleges
– Previously substantial autonomy in IT use and L&T at School level
• Our TEE platform is branded ‘Learning Central’
Learning Central…
• Blackboard 9.1 SP9– Locally hosted– Course delivery system– Content management system– Community engagement system– Blackboard Mobile– Turnitin & Grademark– Learning Objects– Questionmark– Wimba Voice tools– A few custom building blocks
Some (approximate) numbers
• 25,000 FTE equivalent• 6000 staff• 27 schools• 6,000 modules a year• 20,000 visits a day
But what are they all doing?
• Which tools are being used?• And which are not…• Who is using them?• When are they using them?
Assessing our effectiveness
• We implement tools for learning• We deliver staff development• We write support material• We visit schools & talk to staff
• How can we tell if this activity has changed anything?
Our challenges
• Limited supply of resources• Plenty of demand
– For increased administrative efficiency– For improved educational quality / NSS
• Diversity of needs and practice• Always something new
– New features in Service Packs– New tools from suppliers– Emerging topics & changing practice
We need more data!
• Prioritising activities– Avoid spending time on low impact areas
• Identifying areas of low/high uptake– And discovering why
• Assess whether our activity affects things– Evidence for senior managers
• Building business cases– For additional tools or resources– Evidence for deprecating obsolete tools
Google Analytics
• Data on pages visited and when• Page load time• Failure to load• Dwell time• Access to individual tools
• An excellent overview, but usage is amalgamated
Manual reporting
• Auditing every module & assessing against benchmarks
• Extremely flexible in what it reported• Can not be done in real time; a snapshot• Expensive in staff time
Not a practical reporting solution, except as infrequent benchmarking
Blackboard reports
• Excellent tool for assessing the learning activity of individual users
• Data on student’s use of tests, discussion etc• Valuable tool for academics to check progress
of students
Not useful for our purposes as system admins or learning technologists
BIRT
• Uses:– Bb databases– SQL– Eclipse– Building blocks
• Create custom reports run through Building Blocks
• Recently made available
Fairly technical process and runs against live database
Eesysoft Analytics
• Detects where user is in application– Page loads– Mouse movement over HTML elements
• Records these, and associated user role• Reports on
– What different roles have been doing– Trends in use of different tools– Comparisons between tools
How it works
• Admin takes HTML snapshot of page– Define HTML elements
• JavaScript embedded in Bb hotspot• Detects what mouse pointer is over or which
page is loaded• Takes data from user profile• Sent to Eesysoft server & recorded
Contextual help & Eesysoft
• Our entry point to Eesysoft• Provides support to users on the basis of what
they doing at the time– Proactive pop ups for hints/issues– Via Support tab for support materials
• Makes it easier for users to get relevant support in a timely manner
• Multi-language support– Based on language pack in use
Context sensitive help
Pop up help
• Alert users to new functions or problems• Only seen by those who it is relevant to• Only seen at the moment it is relevant to them
EesyAnalytics
• Only in use a short time• Currently for SysAdmin or Learning Techs• Aiming to produce tailored reports for various
people in university• I’ve excluded some data (to improve clarity)• Graphs show %age user group
– Those who’ve accessed, not numbers
What are students using in Schools
• Profile of tool usage in different Schools
What are staff using in Schools…
Undergraduates & postgraduates
Uptake of new tools• Money well spent!
Trendlines
• Pattern of tools over a year• Assessment deadline profile• Change in behaviour after training• Change in use after engagement or awareness
raising
Pattern of student access to Turnitin in two Schools
Language pack & other metrics
• Eesysoft can collect data from user profile• Useful for Welsh language support material• Demographic comparisons… potentially
– Do language pack users use different tools?– Does age make a difference?– Does prior education change what you do?
• Use evidence to build better courses
Roles Roll Call
• Has identified a profusion of roles– 346 in total!– We have a nested structure– And a variety of ad hoc roles
• New Roles– Year of study (staff only)– Year started at University (staff & students)– Will allow us to see how use of changes
What do these allow us to do?
• Focus our efforts better– Training– Support– Engagement, promotion
• Identify areas of good/bad practice– And share lessons– Help academics understand & visualise their students
use of the VLE
Assessing our effectiveness
• Promotions events• Awareness raising• Staff engagement• Using roles to evidence the subsequent
actions after training
Evidence for business cases
• ‘Hard’ evidence to go alongside focus groups and reported needs
• Comparing use of two tools– Use of native and externally supplied
• Highlighting use of a tool– In order to retain it– In order to demonstrate the need for support– To show when something is no longer used
Conclusions
• Beginning of our journey• Benchmarking & strategising• Many data sources are available to us• We can build up a picture of activity in VLE• We can use that to improve our work and
demonstrate our value
• More work to be done and more possibilities to explore
Pete Obee: [email protected]