wg3 interplay between design and evaluation, quality models and standards twintide coimbra, portugal...
TRANSCRIPT
![Page 1: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/1.jpg)
WG3 Interplay between Design and Evaluation, Quality Models and
Standards
Twintide Coimbra, Portugal March, 2013
Ebba Þóra Hvannberg
![Page 2: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/2.jpg)
WG3a Qualities in use
• Identify which software qualities in use (e.g., trust, creativity, automaticity, security, safety, sociability, usability, reliability, evolvability - (related to standards, e.g., ISO 9241, 27477))
– are differentiated by sector-dependent or sector-independent criteria and describe connections
– vary over lifecycle – are traded-off against quality attributes– are prioritized and realized in the process of
systems development in different sectors/disciplines
![Page 3: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/3.jpg)
WG3c design-evaluation feedback
• Understand how iterative design-evaluation-redesign feedback cycles operate for– computing systems in different sectors– relative to specific evaluation criteria: social,
economic, technical etc.– traceability through cycle
![Page 4: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/4.jpg)
Past activities
• Quality attribute maps of relations between domains
– Created initially (Bertinoro 2011)– Refined ( London, 2012)– Consolidated (Skopje, 2012)
![Page 5: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/5.jpg)
Analysis of Transfer of Quality Attributes
• The goal was to strengthen the quality attribute map
• Papers selected with the following criteria– Quality attribute(s), Sector/Domain, Method – Had to have empirical data
• Papers analysed– 9 members contributed to the analysis of 17
papers
![Page 6: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/6.jpg)
Stories of transfer described
• Stories on transfer described
• As a <role> I want to <task> so that I can do <need>
• Storytellers were asked to think about method transfer when describing a task
![Page 7: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/7.jpg)
![Page 8: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/8.jpg)
Prescribed format (from facilitator)
• 20 stories were collected in London, March 2012
• The stories have been classified and cleaned
• A set of 12 stories remain that are related to some kind of transfer of method from one „method application“ to another
![Page 9: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/9.jpg)
Forms of transfer
• Transfer between domains– E-Learning to Games
• Transfer from generalization to specialization– Heuristics evaluation for Virtual Environments
• Adoption of a technique from another field followed by adaptation – E.g. Laddering from marketing
![Page 10: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/10.jpg)
Stories analysed and grouped
• Generalization to specialisation• Specialisation• Transfer between domains• Transfer between similar domains• Adaptation
![Page 11: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/11.jpg)
Open questions
• How can these results be verified or expanded– In the literature, look for similar patterns
• Case studies • Knowledge on transfer • Collect more transfer stories from literature
– In practice• Designers and developers change jobs often• They may be able to tell us transfer stories, i.e. how they
transfer knowledge/methods between sectors
![Page 12: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/12.jpg)
Copenhagen 2012 results
• Tell a story of how you have carried out an evaluation, e.g. in a project and contrast it to another instance of an evaluation.
• 18 cases of transfer stories were collected from Twintide members
• Results are here
![Page 13: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/13.jpg)
Overview of form of transfer
• Overview
![Page 14: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/14.jpg)
Revising stories and further analysis
1. Read and amend story1. Typos2. Specify domain of story 3. Review stories4. Review transfer between stories 5. Add publication if available
![Page 15: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/15.jpg)
Identify method resources used in the stories *
1. Participant-recruitment 2. Task selection3. Reporting format4. Problem identification5. Problem classification6. Analysis7. Heuristics8. Thinking aloud protocol 9. Please add more as needed
*Law, E.L.-C., Hvannberg, E., Vermeeren, A.P.O.S., Cockton, G., Jokela, T.: Made for Sharing: HCI Stories of Transfer, Triumph and Tragedy. CHI 2013 Extended Abstracts. ACM, Paris, France (2013)**Woolrych, A., Hornbæk, K., Frøkjær, E., Cockton, G.: Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes. International Journal of Human-Computer Interaction 27 (2011) 940-970
![Page 16: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/16.jpg)
Identify contextual factors*
1. Business goal2. Design purpose3. Development context4. Team skills5. Corporate culture/values6. Please add more as needed
*Law, E.L.-C., Hvannberg, E., Vermeeren, A.P.O.S., Cockton, G., Jokela, T.: Made for Sharing: HCI Stories of Transfer, Triumph and Tragedy. CHI 2013 Extended Abstracts. ACM, Paris, France (2013)**Woolrych, A., Hornbæk, K., Frøkjær, E., Cockton, G.: Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes. International Journal of Human-Computer Interaction 27 (2011) 940-970
![Page 17: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/17.jpg)
Contrast Method resources
Item Story 1 Story 2 Contrast/ Type of transfer
Participant recruitmentTask selection
etc.
![Page 18: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/18.jpg)
Contrast Contextual Factors
Item Story 1 Story 2 Type of transfer
Business goal
Design purpose
Development contextTeam skills
Corporate culture
![Page 19: WG3 Interplay between Design and Evaluation, Quality Models and Standards Twintide Coimbra, Portugal March, 2013 Ebba Þóra Hvannberg](https://reader035.vdocuments.mx/reader035/viewer/2022070407/56649e435503460f94b35da9/html5/thumbnails/19.jpg)
Schedule
Task Duration Time
Presentation of previous results
30 minutes 14:00
Read and amend stories 30 minutes 14:30
Identify method resources and contextual factors
30 minutes 15:00
Coffee break 30 minutes 15:30
Contrast stories 30 minutes 16:00
Report method resources and contextual factors
30 minutes 16:30