tool criticism

Download Tool Criticism

Post on 03-Aug-2015




0 download

Embed Size (px)


1. Tool Criticism Myriam C. Traub, Jacco van Ossenbruggen, Lynda Hardman! Information Access 2. Amsterdam 1642 Amfterdam 1624 ? 16xx? 2 First mention of 1618 in the OCRed newspaper archive of the KB? Amslerdam 1673 earliest ! document 3. 01 Digital Humanities digital archives / large data collections! collection bias / digitization policy! digital representation physical object! tools are imperfect ! source cannot be considered independent from tools! need methods to detect tool- induced bias 3 4. 01 Source criticism well established method in the humanities to detect bias in a source! Who is the author?! Is the information current?! Is the information objective and credible?! need a similar approach for tool-induced bias: ! ! tool criticism 4 5. 01 Methodology interviews with humanities scholars! classication of common research tasks! lack of trust blocks progress! use case: digital newspaper archive of KB ! no formal OCR evaluation! useful for scholars?! mismatch between two perspectives 5 6. We care about average performance on representative subsets for generic cases. I care about actual performance on my non- representative subset for my specic query. 6 Two different perspectives of quality evaluation 7. No silver bullet we propose novel strategies that solve part of the problem:! critical attitude ! (awareness and better support)! transparency ! (provenance, open source, documentation, )! alternative quality metrics! (taking research context into account) 7 M. Traub, J. van Ossenbruggen, L. Hardman, ! Impact Analysis of the OCR Quality Problem in Digital Archives, TPDL2015 (under review) 8. 8 Amsterdam Context-aware quality indicators query data task 9. 01 Future work What strategies should a tool support to help scholars discover and dealing with bias?! What is a good way of estimating uncertainty for a specic task?! Can we crowdsource (part of the data for) better estimates?! What is a good way of conveying the estimated impact to scholars? 9 questions?