iterative usability evaluation of dsls

Download Iterative usability evaluation of DSLs

Post on 18-Jul-2015

92 views

Category:

Education

4 download

Embed Size (px)

TRANSCRIPT

  • Ankica Barii, PhD student

    Supervisors: Vasco Amaral, Miguel Goualo

  • A language is a means of communication

    The user interface is a realization of a languageA language is a model that describes the allowed terms and how to compose them into valid sentences

    *

  • General Purpose (programming) Languages (GPLs)User has to master programming concepts User has to master domain concepts

    Domain Specific (modeling) Languages (DSLs)Meant to close gap between PROBLEM DOMAIN and SOLUTION DOMAINReduce the use of computation domain conceptsFocus on the domain concepts*

  • Increasingly popularRaise the abstraction level (closer to the domain)Narrow the design space

    Several benefits claimed, in well-defined domainsProductivity gainsBetter time to marketAvoid error-prone mappings between domain and software development concepts Leverage the expertise of domain experts*

  • [Mernik, 2005]*REUSE?

  • a model of theUser Different knowledge setsCharacteristics chosen are dependent on application domainthe hardware-softwarePlatformset of computing, sensing, communication, and interaction resources e.g. operating systems, memory size, network bandwidth, input and output interaction devicesthe social and physicalEnvironmentWhere the interaction is actually taking placeDifferent languages may have different contexts of useTheir users are likely to have different knowledge setsA minimum set of ontological concepts is required to use the language*

  • The user's view of the Quality of a productMeasured in terms of the result of using the product, rather than its properties

    *

  • In general, software language engineers DO NOT evaluate their languages with respect to their impact in the software development process in which the DSLs will be integratedOr, if they do, they are extremely shy about it*

  • *

  • What to measure?

    *

  • *[Barisic, 2011c]

  • *Introduce DSLs Usability evaluation during DSLs life-cycle iterations Design an effective experimental evaluation of DSLs that will provide qualitative and quantitative feedback for DSLs developersProduce user-centered design of DSLForesee the Quality of a DSL while in an iterative evolution stepMerge the Software Language development process with the Usability Engineering process

  • *Barii, Monteiro, Amaral, Goulo, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012

  • *Barii, Monteiro, Amaral, Goulo, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012

  • *Barii, Monteiro, Amaral, Goulo, Monteiro: "Patterns for Evaluating Usability of Domain-Specific Languages, InProceedings of the 19th Conference on pattern languages of programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012

  • *Language engineerDomain expertUsability engineer

  • *

  • per group*

  • Language goals: Support users in creating and manipulating Evaluation:Compare two different DSL solutions within same domain sharing the same goalFor the purpose of having means to reason about what is more usable: Abstract syntax definitionTransformation Interaction

    *

  • Development environment and methodology: EpsilonModel-Driven developmentDomain: Management and organizationTransformation output: MS Excel templatePlace:Summer school on MDD of DSLs, Faculty of Organizational Science's

    *

  • Domain Experts: subjectsUsability Experts: experiment designersLanguage Developers: lecturers, subjectsEnd Users: subjects

    *

    LecturersExperiment designersSubjects2239

  • *Instrument: background questionnaire

  • Our evaluation technique was tested with two individuals (two physics experts) in order to verify it and to test the teaching materials and questionnairesAs time constrants and equipment turn out to be adequat there was no need to change prepared materials*[Howatt 1995]

  • RQ1:Is querying with Pheasant more effective than with C++/BEE?RQ2:Is querying with Pheasant more efficient than with C++/BEE?RQ3:Are participants querying with Pheasant more confident on their performance than with C++/BEE?

    Our goal is to:analyze the performance of Pheasant programmers plug-insfor the purpose of comparing it with a baseline alternative (C++/BEE)with respect to the efficiency, effectiveness and confidence of defying queries in Pheasantfrom the point of view of a researcher trying to assess the Pheasant DSL,in the context of a case study on selected queries*Instrument: background questionnaire

  • H1null Using Pheasant vs. C++/BEE has no impact on the effectiveness of querying the analysis frameworkH1alt Using Pheasant vs. C++/BEE has a significant impact on the effectiveness of querying the analysis framework

    H2null Using Pheasant vs. C++/BEE has no impact on the efficiency of querying the analysis frameworkH2alt Using Pheasant vs. C++/BEE has a significant impact on the efficiency of querying the analysis framework

    H3null Using Pheasant vs. C++/BEE has no impact on the confidence of querying the analysis frameworkH3alt Using Pheasant vs. C++/BEE has a significant impact on the confidence of querying the analysis framework*

  • We focus on presenting six examples, each focusing in some of the features we chose to evaluateParticipants are asked to give themselves a mark for feeling of correctness of their trialSession take the time needed for each group to understand the examples*

  • Every participant has four queries, specified in English, to be rewritten in previously learned languageSubjects makes self-assessment of his replay rating his feeling of correctness

    Example:Build the decay of a D0 particle to a Kaon Pion*

  • Results obtained with Pheasant were clearly better then those with C++/BEEPheasant allowed non-programmers to correctly define their queries.The evaluation also showed a considerable speedup in the query definition by all the groups of users that were using PheasantThe feed-back obtained from the users was that it is more comfortable to use Pheasant than with the alternative.*

  • The participants were asked to judge the intuitiveness, suitability and effectiveness of the query language. The goal was to evaluate:Overall reactions Query language constructsAffect to query language was rated by:Query language constructsParticipants comments*

  • *Foster higher productivity of DSL end usersIncreased quality in use and quality of experience while using a DSLReplication of experimentsReusing data and recommendations Reduce complexityTraceability of usability recommendations and design changesReduced cost of evaluation, development and supportIncreased reliability of usability evaluation results

  • *

  • *

  • *Foster higher productivity of DSL end usersIncreased quality in use and quality of experience while using a DSLReplication of experimentsReusing data and recommendations Reduce complexityTraceability of usability recommendations and design changesReduced cost of evaluation, development and supportIncreased reliability of usability evaluation results

  • *

  • Literature[Mernik2005] M. Mernik, J. Heering, and A. M. Sloane: When and how to develop domain-specific languages, 2005, ACM Computing Surveys[Gabriel2010] Gabriel, P., Goulo, M. & Amaral, V. (2010). Do Software Languages Engineers Evaluate their Languages? in XIII Congreso Iberoamericano en "Software Engineering" (CIbSE'2010)[Barisic2011a] Barii, A., Amaral, V., Goulo, M., and Barroca, B.: Quality in Use of DSLs: Current Evaluation Methods. Proc. 3rd INForum - Simpsio de Informtica (INForum2011), Coimbra, Portugal, September 2011[Barisic2011b] Barii, A., Amaral, V., Goulo, M., and Barroca, B.: Quality in Use of Domain Specific Languages: a Case Study. Proc. Evaluation and Usability of Programming Languages and Tools (PLATEAU) Portland, USA, October 2011 [Barisic2011c] Barii, A., Amaral, V., Goulo, M., and Barroca, B.: How to reach a usable DSL? Moving toward a Systematic Evaluation, Electronic Communications of the EASST, 2011[Barisic2012] Barii, A., Amaral, V., Goulo, M., and Barroca, B.: Evaluating the Usability of Domain-Specific Languages, in Mernik, M. (Ed.): Formal and Practical Aspects of Domain-Specific Languages: Recent Developments (IGI Global, 2012)

    *

  • **********************