assessing the usability of machine translated content: a user-centred study using eye tracking dr....
TRANSCRIPT
![Page 1: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/1.jpg)
Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking
Dr. Stephen Doherty & Dr. Sharon O’BrienCentre for Next Generation Localisation
School of Applied Language & Intercultural Studies
Dublin City University
![Page 2: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/2.jpg)
Outline
Introduction
Research Aims
Methods
Results
Conclusions
![Page 3: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/3.jpg)
Introduction
Increased need for translation
Diversity of content and users
Rise in prevalence of machine translation [MT] both off- and online
Mixed reports of quality – attitudes and expectations
Divergence in R&D – translation studies/computer science
Evaluation metrics – human and automatic
Our focus here is on usability
![Page 4: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/4.jpg)
Research Aims
To investigate if there are differences in usability between the English [source language] and the unedited machine translated target languages [FR, DE, SP, JP].
Or in other words: how usable is machine translated content?
Adoption of the ISO/TR 16982 definition of usability
Importance of ecological validity: real materials and users
![Page 5: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/5.jpg)
Methods
User-centred approach [n = 30]; task driven – ‘new user’ scenario
Eye tracking [tobii 1750]:Fixation count and average duration
Attentional shifts; percentage time in each window
Textual regressions
![Page 6: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/6.jpg)
![Page 7: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/7.jpg)
Methods
Post-task questionnaire; five-point LikertComprehension
Task completion
Potential improvement
Future reuse
Recommendation
Recall
![Page 8: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/8.jpg)
Methods
UsabilitySatisfaction
Efficiency [task success/task time]
![Page 9: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/9.jpg)
Eye Tracking
Task timeLowest for EN [sig. JP]
Fixation count and average durationLowest for EN [sig. JP] for both
Attentional shifts; percentage time in each windowEN and FR spent most time in task window
EN fewest shifts of attention [sig. JP]
Textual regressionsRaw number and distance: EN and SP [sig. JP]
‘Long’ regressions: JP [sig. all others]
![Page 10: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/10.jpg)
Questionnaire Results
ComprehensionEN rated highest [sig. for FR and JP]
Task completionEN rated highest [sig. for JP]
Potential improvementSP & EN rated as needing least improvement, but could still be improved upon
Future reuseFR & EN rated highest
RecommendationEN rated highest [sig. for JP and DE]
RecallEN scored highest [sig. for JP and DE]
![Page 11: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/11.jpg)
Usability Results
SatisfactionEN rated highest [sig. for FR, DE, and JP]
Task completionEN and SP more successful [sig. JP]
EfficiencyEN most efficient [sig. JP and DE]
![Page 12: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/12.jpg)
Conclusions
So, just how usable is raw MT?Similar results for EN, SP, and FR
DE and JP more problematic [MT system]
Functionally usable [more than just ‘gisting’]
UX best for EN users
MT viable for certain pairs
Human intervention necessary to ensure best UX
![Page 13: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/13.jpg)
Questions?
[email protected] [email protected]
This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (www.cngl.ie) at Dublin City University.
![Page 14: Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon OBrien Centre for Next](https://reader035.vdocuments.mx/reader035/viewer/2022062618/5513d8965503466f748b505d/html5/thumbnails/14.jpg)
Predictors of Positive UX
Satisfied users: comprehension & task time
Satisfied users: recommend to others
Task completion: textual regressions
Cognitive effort: instructions aiding task completion