UPTEC IT 14 012

Examensarbete 30 hpSeptember 2014

Integration of Quantitative User Data Into the Agile Website Development Process

Emma Rangert

Teknisk- naturvetenskaplig fakultet UTH-enheten Besöksadress: Ångströmlaboratoriet Lägerhyddsvägen 1 Hus 4, Plan 0 Postadress: Box 536 751 21 Uppsala Telefon: 018 – 471 30 03 Telefax: 018 – 471 30 00 Hemsida: http://www.teknat.uu.se/student

Abstract

Integration of Quantitative User Data Into the AgileWebsite Development Process

Emma Rangert

Website analysis with Google Analytics takes place by the end of the developmentprocess when performed at Valtech AB. Measurements are added to gatherquantitative user data from websites when the website is live instead of as a part ofthe development process. Through interview sessions and a literature study the LeanStartup philosophy and lean user experience have been studied with focus on theminimum viable product, hypothesis writing, and the build-measure-learn feedbackloop. This thesis work proposes a process to integrate the work of website analysisthrough Google Analytics into the agile work process. The proposed process includesearly definition of business impact, key performance indicators and main conversionsof the website. Also early set up of the Google Analytics accounts and a dashboardscreen monitoring the main macro conversions are an important part. Furthermore,hypothesis writing and creation of minimum viable products as well as considerationof previous measurements in backlog prioritization and refinement meetings areincluded. Finally, continuous presentation of data measurements at the sprint demo isimportant. The conclusions of this thesis work are that the motivation for performingagile website analysis depends on the development team members and theirknowledge of website analysis. It is also important that the customer is active in thesense of learning and taking part in the measurement process to be able to take overthe analysis at project completion. Lastly, it is important to follow up quantitativemeasurements with qualitative measurements by asking questions why the user actedin a certain way.

Tryckt av: Reprocentralen ITC ISSN: 1401-5749, UPTEC IT 14 012Examinator: Olle ErikssonÄmnesgranskare: Roland BolHandledare: Christian Wigren

Popular Scientific Summary in Swedish

I dagens webbutveckling ligger mycket fokus pa anvandbarhet och att generellt skapa enangenam upplevelse for anvandaren, samtidigt som agarens mal med webbsidan uppfylls.Malet med en webbsida kan vara att salja varor eller tjanster det vill saga e-handel, mendet kan ocksa vara av mer svavande karaktar som att oka allmanhetens kannedom omnagonting.

For att mata anvandarnas beteende och i vilken omfattning webbsidans mal uppfyllskan man anvanda analysverktyget Google Analytics. Google Analytics fungerar som ettmatverktyg dar matpunkter skapas genom att sma avsnitt med programkod laggs in iwebbsidans kallkod. Darefter samlas statistik i ett webbaserat anvandargranssnitt hosGoogle under tiden som anvandarna besoker webbsidan.

Valtech AB ar ett IT-foretag som dagligen utvecklar webbsidor at sina kunder. Deanvander Google Analytics for att gora webbanalys, det vill saga samla kvantitativanvandardata och analysera den for att fatta beslut. Problemet ar att webbanalysenofta kommer in sent i projekten och kanske rentav i slutet da projektet lamnas over tillkunden. Detta resulterar i att programkoden for matningen blir svarare att lagga in,datainsamlingen blir inte heltackande och det blir dyrare att genomfora webbanalysen.Annu varre ar att man kanske upptacker att man har tankt helt fel tidigt i systemutveck-lingen, for nu visar statistiken om anvandarnas beteende att en viss funktion inte allsanvands sa som man hade tankt eller trott.

Malet med det har examensarbetet har varit att foresla en process for hur Valtech bor arb-eta for att fa med webbanalys som en kontinuerlig del av det agila utvecklingsarbetet avwebbsidor. Agil systemutveckling ar en samling av metoder som bygger pa gemensammaprinciper om flexibilitet och lattrorlighet. Arbetssatten syftar till att ha nara och regel-bunden kontakt med kunder och anvandare for att pa sa satt gora dem annu nojdare.Agil systemutveckling innebar ofta att systemutvecklarna arbetar i korta perioder, exem-pelvis tva veckor, och levererar ett fungerande delsystem efter varje period. Exempel papopulara agila arbetssatt ar Scrum och Kanban, vilka ocksa tas upp i detta arbete.

Under arbetets gang har begreppen Lean Startup och Lean UX studerats med fokuspa de ingaende delarna minimal gangbar produkt (eng. minimum viable product), hy-potesskrivning samt bygg, mat och lar feedback-loopen. En minimal gangbar produkt ardet minsta man behover utveckla for att kunna testa en hypotes eller tanke. En litterat-urstudie i kombination med intervjuer har genomforts for att undersoka hur dessa begreppkan anvandas for att losa problemet hos Valtech. Efter att ett forsta processforslag tagitsfram utvarderades det genom intervjuer med projektmedlemmar och kundrepresentanter.

Den process som arbetet resulterat i innefattar att tidigt bestamma de mal for webbsidansom ska uppfyllas och vilka matpunkter som behovs for att mata dem. Vidare ar detviktigt att strukturen for Google Analytics kontot satts upp i borjan av projektet. For atthela projektgruppen och aven kundrepresentanter enkelt ska kunna ta till sig information

om hur det gar med maluppfyllelsen, bor en fysisk bildskarm som visar de viktigaste malensattas upp. Under systemutvecklingsprocessen bor man skriva hypoteser som beskrivervad man tror att anvandarna vill ha och hur det ska matas. Utifran hypoteserna bor mansedan utveckla minimala gangbara produkter for att testa att utvecklingen ar pa vag atratt hall. Tidigare matningar fran det nuvarande projektet, men aven tidigare projekt,bor anvandas for att fatta beslut om hur man ska ga vidare i projektet.

Det ar viktigt att uppna en miljo dar samtliga nyckelpersoner, sa val systemutvecklaresom bestallare och kunder, tycker att matningarna ar viktiga och nodvandiga. Detta borgoras genom att regelbundet presentera matningar vid de tillfallen da systemet visas uppfor kunder och bestallare.

Utifran examensarbetet kan slutsatsen dras att i forsta hand fokusera pa att oka med-vetenheten och delaktigheten i webbanalysen hos alla anstallda pa Valtech. Detta efter-som webbanalysen genom den har processen kommer att bli en del av deras dagliga arbete.Om ett okat intresse for att arbeta med webbanalys kan uppnas, kommer det heller inteatt vara nagra problem att uppna agil webbanalys. Dock kravs det mer for att helt gaover till en Lean Startup eller Lean UX organisation eftersom det kraver ganska storaorganisationsforandringar for ett foretag beroende pa hur organisationen ser ut idag.

Acknowledgments

First of all I would like to thank my supervisor at Valtech, Christian Wigren. Thankyou for your help and guidance, all the time spent on discussing issues and ideas, for theweekly meetings, and for providing me with constructive and encouraging feedback.

I would also like to thank my reviewer Roland Bol for helping me with language andwriting issues, and for providing guidance on the methods I chose to use. Thank you foralways being able to answer my questions.

A special thank you goes to my friend Carl Ekman who helped me graphically visualizethe Scrum process. Finally, I would like to thank all the employees at Valtech whoparticipated in the pre-study interviews and the evaluation sessions, as well as spent timeanswering the questionnaire. Without your help I would have missed out on valuableinformation.

Contents

Glossary 9

Acronyms 10

1 Introduction 111.1 Problem Description and Motivation . . . . . . . . . . . . . . . . . . . . 111.2 Purpose and Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.3 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.4 Structure of the Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2 Background 142.1 Website Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.1.1 Key Performance Indicator . . . . . . . . . . . . . . . . . . . . . . 142.1.2 Analytics and Ethics . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.2 Agile Software Development . . . . . . . . . . . . . . . . . . . . . . . . . 172.3 Scrum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.3.1 The Scrum Team . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.3.2 Scrum Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.3.3 Scrum Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.4 Kanban . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202.5 Agile at Valtech AB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.6 Agile Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.7 Integration of UX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3 Theory 243.1 The Lean Startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.1.1 Validated Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 243.1.2 Minimum Viable Product . . . . . . . . . . . . . . . . . . . . . . 243.1.3 Build Measure Learn . . . . . . . . . . . . . . . . . . . . . . . . . 243.1.4 Pivot or Persevere . . . . . . . . . . . . . . . . . . . . . . . . . . 253.1.5 Lean Startup Applied to Mobile Application Development . . . . 26

3.2 Lean UX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.2.1 Principles of Lean UX . . . . . . . . . . . . . . . . . . . . . . . . 273.2.2 The Hypothesis and MVP . . . . . . . . . . . . . . . . . . . . . . 273.2.3 Suggested Organization Shifts . . . . . . . . . . . . . . . . . . . . 28

3.3 Business Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.3.1 Business Impact Management . . . . . . . . . . . . . . . . . . . . 283.3.2 Business Impact Mapping . . . . . . . . . . . . . . . . . . . . . . 29

3.4 Google Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.4.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.4.2 Dashboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.4.3 Custom Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.5 Theory and Research Questions Connected . . . . . . . . . . . . . . . . . 31

4 Methods 32

4.1 Literature Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.2 Google Analytics Implementation . . . . . . . . . . . . . . . . . . . . . . 324.3 Observation of an Agile Software Development Team . . . . . . . . . . . 324.4 Pre-study Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.4.1 Interview Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.4.2 The Interviewees . . . . . . . . . . . . . . . . . . . . . . . . . . . 334.4.3 Interview Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.5 Evaluation of Proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354.5.1 Participating Projects . . . . . . . . . . . . . . . . . . . . . . . . 354.5.2 Questionnaire Regarding Project Measurability . . . . . . . . . . 364.5.3 Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.5.4 Inspection and Adaption of Product Backlog . . . . . . . . . . . . 36

5 Pre-study Interview Summary 375.1 User Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375.2 Business Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375.3 Design Driven by Statistics and Learning . . . . . . . . . . . . . . . . . . 385.4 Team Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

6 The Proposed Process 416.1 Project Pre-study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416.2 Project Startup - Sprint 0 . . . . . . . . . . . . . . . . . . . . . . . . . . 426.3 During Ongoing Project - Sprint 1 to Sprint N . . . . . . . . . . . . . . . 426.4 Project Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

7 Evaluation 457.1 Questionnaire Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

7.1.1 Team perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . 457.1.2 Customer Perspective . . . . . . . . . . . . . . . . . . . . . . . . . 47

7.2 Summary of Interview Answers . . . . . . . . . . . . . . . . . . . . . . . 487.2.1 General Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . 487.2.2 Previous Experience of Measurement . . . . . . . . . . . . . . . . 487.2.3 Importance of KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . 497.2.4 Dashboard Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . 497.2.5 Hypothesis Writing and MVP . . . . . . . . . . . . . . . . . . . . 497.2.6 Backlog Refinement Session . . . . . . . . . . . . . . . . . . . . . 507.2.7 Presentation of Measurements at Demo . . . . . . . . . . . . . . . 50

7.3 Conclusion of Improvements . . . . . . . . . . . . . . . . . . . . . . . . . 51

8 Discussion 528.1 Research Questions Revisited . . . . . . . . . . . . . . . . . . . . . . . . 52

8.1.1 Use of Measurements Between Sprints . . . . . . . . . . . . . . . 528.1.2 Team Composition . . . . . . . . . . . . . . . . . . . . . . . . . . 528.1.3 The Issue of Early Release . . . . . . . . . . . . . . . . . . . . . . 53

8.2 The Proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548.2.1 General Di�culties . . . . . . . . . . . . . . . . . . . . . . . . . . 548.2.2 Analysis Already Being Performed . . . . . . . . . . . . . . . . . 54

8.2.3 Possible Obstructions . . . . . . . . . . . . . . . . . . . . . . . . . 558.2.4 Alternative Solutions . . . . . . . . . . . . . . . . . . . . . . . . . 55

8.3 Method Criticism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568.3.1 The Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568.3.2 The Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568.3.3 A Status Inventory . . . . . . . . . . . . . . . . . . . . . . . . . . 578.3.4 Compliance With the Agile Manifesto . . . . . . . . . . . . . . . . 57

8.4 Website Analysis and Ethics . . . . . . . . . . . . . . . . . . . . . . . . . 57

9 Conclusions 589.1 People and Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . 589.2 Lean Startup, Lean UX, and Agile Website Analysis . . . . . . . . . . . . 589.3 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589.4 A Minimum Viable Process . . . . . . . . . . . . . . . . . . . . . . . . . 599.5 Recommendations for Valtech . . . . . . . . . . . . . . . . . . . . . . . . 599.6 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

References 60

A Pre-study Interview Questions 64A.1 General Information Interviews . . . . . . . . . . . . . . . . . . . . . . . 64

A.1.1 Agile and General Work at Valtech . . . . . . . . . . . . . . . . . 64A.1.2 User tests at Valtech . . . . . . . . . . . . . . . . . . . . . . . . . 64

A.2 Project Manager Interviews . . . . . . . . . . . . . . . . . . . . . . . . . 65A.3 Web Analyst Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . 66A.4 Software Developer Interviews . . . . . . . . . . . . . . . . . . . . . . . . 67

B Questionnaires 68B.1 Team Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68B.2 Customer Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68B.3 Evaluation Interview Questions . . . . . . . . . . . . . . . . . . . . . . . 69

C The Complete Process 70C.1 Project Pre-study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70C.2 Project Startup - Sprint 0 . . . . . . . . . . . . . . . . . . . . . . . . . . 70C.3 During Ongoing Project - Sprint 1 to Sprint N . . . . . . . . . . . . . . . 70C.4 Project Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Glossary

A/B testing is a form of statistical hypothesis testing where two versions, A and B,are compared. A and B are identical except for one variation that might a↵ect theuser’s behavior. 12, 16, 27, 46

Automated test also known as test automation, is the process of writing test scriptsand use of another software to test the developed software. Test automation is usedto run test scenarios quickly and repeatedly and is a step of successful continuousintegration. 43, 70

Beta version is a pre-release of software that is given out to a large group of users totry under real conditions. Software in the beta phase will generally have many morebugs than completed software. The users are usually expected to report any bugsthey encounter or changes they like to see before the final release. 53

Continuous deployment is the design practice used in software development to auto-mate and improve the process of software delivery. In other words it is the deploy-ment or release of code to a production environment once the developer considersthe code ready to ship. The idea is to release the code to a user base. 12, 31, 39,43, 49, 52, 57, 58, 70

Continuous integration in software engineering, is the practice of merging developersworking copies with a shared mainline several times a day. The idea is to test thecode as often as possible to catch issues early. 12, 43, 70

Conversion is a completion of an activity on a website that is important to the successfor some business. In other words, any action that the website owner considers tobe the objective. 14, 22, 30, 39, 41, 53, 69

GitHub is a web-based service for version control of software. 49

Increment is the sum of all the product backlog items completed during a sprint andthe value of all the previous sprints. 11, 18, 31, 41

Kanban is a scheduling system for lean and just-in-time production, developed by Taii-chi Ohno at Toyota. 12, 13, 17, 20–22, 54, 58

Lean manufacturing is a production practice that considers the expenditure of re-sources for any goal other than the creation of value for the end customer to bewasteful, and thus a target for elimination. 24

Macro conversion is the primary conversion point of a website, for example a purchaseat an e-commerce site. 39, 42, 69

Micro conversion is one of several activities that a user do before making a macroconversion, for example an email sign up. 39

9

PhantomJS is a headless browser that can be programmed using JavaScript. It hasbuilt-in rendering functionality and thus can take screen shots of visited pages. 42,70

Scrum is a an iterative and incremental agile software development framework for man-aging software projects, developed by Je↵ Sutherland and Ken Schwaber. 11–13,17–23, 26, 31, 32, 47, 52, 54, 55, 58, 70

Selenium is a framework for software testing for web applications. It provides recordtools authoring tests without learning a test scripting language. Thus it is possibleto write tests in Java and C# to name a few. 42, 47, 70

Sprint 0 is a sprint held before the actual start of the project, in this sprint activitiessuch as setting up necessary technology and logistical issues such as workplace forthe team are performed. 42, 69, 70

Task is a piece of work that is a part of a user story in agile development methodologiessuch as Scrum. 20, 22, 40, 51

User story in software development and product management, a user story is a fewsentences describing what a user of the system does or needs to do as part of theirjob function. Often formulated on the format ”As a [role], I want [desire] so that[benefit]”. 19, 20, 22, 35, 36, 39, 40, 42, 43, 48, 49, 70

Vanity metrics are data collected about a company or its users that do not help en-trepreneurs make decisions. Many claim that these metrics serve no purpose otherthan to make the entrepreneur feel good. 24

Acronyms

ABO Acquisition Behavior Outcomes. 14

KPI Key Performance Indicator. 14, 15, 23, 41, 42, 48, 53, 69

MVP Minimum Viable Product. 12, 24, 25, 27, 43, 48, 49, 52–54, 57, 70

MVT Multivariate Test. 12

UX User Experience. 23, 26–28, 33–35, 37, 39, 44, 46, 51, 56, 57

10

1 Introduction

1.1 Problem Description and Motivation

Valtech AB is a company located in Stockholm, Sweden, that specializes in digitalstrategies as well as web and mobile development. Therefore the company works a lotwith web strategy and tools such as Google Analytics [1] to gather quantitative user dataand to follow-up on customer specified goals. Valtech works solely with agile method-ologies in their system development projects and in their work with digital partnershipwhere further development and maintenance of software are part of the work performed.

Quantitative user data is gathered by measuring user behavior on websites. Today Valtechlacks a work process to integrate such data measurements in the agile work flow at anearly state in the product development process. The product in this sense is the websitebeing built during the development process.

The hypothesis is that the integration of data in the agile work process will improvethe product developed in the project, since user data will be analyzed and made use ofearly in the project. It is possible to discover and remedy flaws in the design and userexperience of the website with this workflow. In addition, to add web analysis at the endof a project or several months after the release of the product is an expensive and di�culttask. Thus the hypothesis include that this will be avoided with integrated web analysisin the agile work flow. Lastly, it will also improve the work process of web strategistsand analysts at Valtech.

As visualized in Figure 1, traditional product development consists of a pre-study followedby a larger development phase. This is also the case in Scrum-based product developmenteven though the development is done in an iterative manner with shorter increments.

Figure 1: Traditional product development

1.2 Purpose and Goal

The purpose of this thesis work is to find a suitable way for Valtech to improve theirwork with website analysis in the products they develop. The main question is ”How canthe work of collecting and analyzing quantitative user data from websites be integrated inthe agile work process at Valtech?”.

The goal of this thesis is to propose a process or checklist to follow to solve the lackof integration mentioned in Section 1.1. In other words: what should the company do

11

at project startup and during the rest of the project to integrate the quantitative data,i.e. the user behavior, from the beginning and throughout the whole project, how shouldquantitative data from the website be used in for example backlog refinement meetingsin Scrum, how should data from measurements be synchronized between sprints, howshould new measurements be compared to old measurements, are example of issues thatneed to be addressed and analyzed in order to propose such a process.

The following research questions have been identified and will be answered throughoutthe thesis work:

• How should measurements be used between sprints?

• How should the agile team be composed?

• How could early release to production be achieved?

With the outcome of the project at hand, quantitative user data from websites underconstruction and analysis of it will be integrated in the agile software development processinstead of taking place after the project is completed as is the case today. This will lead toshorter feedback-loops and it will also go well with continuous integration and continuousdeployment which are very topical at Valtech today. Figure 2 shows an example of whatthis process could look like as opposed to Figure 1. In this process analysis such as A/Btesting, user tests and Multivariate Tests (MVTs) are performed as well as insertion ofMinimum Viable Products (MVPs) into the process.

Figure 2: MVP and lesson-learned driven development

1.3 Delimitations

The quantitative user data that is to be integrated into the agile work flow refer todata gathered through tools such as Google Analytics. Whenever agile frameworks arementioned, it will refer to Kanban or Scrum. The use of Scrum is more widespread thanthe use of Kanban at Valtech and therefore this thesis will focus more on integrating thequantitative data into the Scrum process than into the Kanban process.

12

1.4 Structure of the Report

The rest of the report is partitioned into eight sections with the following content. Sec-tion 2 introduces website analysis and agile software development methods such as Scrumand Kanban.

The result of the research and literature study performed during the thesis work is presen-ted in Section 3. The section describes the Lean Startup philosophy as well as the LeanUX principles. It also addresses business impact in terms of business impact managementand business impact mapping. In continuation it gives a brief introduction to the toolGoogle Analytics.

Section 4 describes the methods and approaches used during the thesis work. It detailsthe interview setup performed both during the pre-study and the evaluation phase of thethesis work.

The result of the pre-study interviews is compiled in Section 5. Section 6 describes theinitial process proposed by the thesis work divided into four project phases: the pre-studyphase, the startup phase, the ongoing phase, and the completion phase.

The evaluation outcomes gathered during the thesis work is described in Section 7. Thesection compiles both the questionnaire results and the interview answers. The sectionis concluded with the improvements resulting from the evaluation.

Section 8 discusses the work performed and answers the research questions. It discussesthe proposal and possible obstructions to it, as well as the used methods. The sectionalso briefly discusses the ethical topic of website analysis.

The conclusions of the thesis work are presented in Section 9. The section also includessome recommendations for Valtech and future work.

13

2 Background

2.1 Website Analysis

In website analysis the most important question to ask oneself is ”why does your websiteexist?” [2]. To manage websites is a continuous task and in order to make the userexperience as good as possible it is necessary to understand how users reached the websiteand how to increase relevant tra�c on the website. By analyzing web tra�c it is possibleto do business and market research, to track where visitors come from, what day is themost suitable to publish a blog post, and to pinpoint how to best spend a marketingbudget to name a few [3].

To analyze a website and the behavior of its users incorporates defining goals and KeyPerformance Indicators (KPIs), analyzing data and take action on it. Web analytics canbe defined as follows: the act of increasing a website’s persuasion and relevancy to achievehigher conversion rates, where persuasion is the impact on the website’s business [2]. Theo�cial definition of web analytics read out:

”Web Analytics is the measurement, collection, analysis and reporting of In-ternet data for the purposes of understanding and optimizing Web usage”.

(Web Analytics Association, 2008 [4])

A web analysis process should include at least five steps: define goals, build KPIs, collectdata, analyze data, and implement changes. The last two steps will be repeated. Further-more, web analytics uses statistics, data mining, and a methodological process to findactionable insights [2]. Kaushik [5] describes the complete customer-business journey asAcquisition Behavior Outcomes (ABO) where acquisition means what is done to attracttra�c to the website, behavior is what happens when people land on the website, andoutcomes is the same as conversions.

Figure 3 shows how web analytics can influence a website’s conversion rate. The websiteof the top-left funnel has a high number of visitors but is unsuccessful in making themconvert, i.e. the persuasion is poor, and the bottom-right funnel represents a websitewith a high number of visitors and a high conversion rate. In other words, the bottom-right website is the best one [2]. The top-right website manages to attract visitors butis ine�cient in persuading them to convert. The bottom-left website has an improvedpersuasion rate, however it still fails to make people convert.

2.1.1 Key Performance Indicator

Key Performance Indicators (KPIs) can be used in order to measure the goal of thewebsite. They show if the website is getting closer to achieving its objectives. Thereshould be an action on the website linked to each defined KPI. Good KPIs should havefour properties: un-complex, relevant, timely, and instantly useful. Un-complex meansthat everyone working with the KPI should understand it, not just the analyst; relevant

14

Figure 3: Customer life cycle funnel [2]

means that the KPI should be relevant to the business it assesses, since each businessis unique; timely means that the information necessary to make decisions needs to begathered on time, and instantly useful means that anyone should understand what theKPI is quickly in order to get insight from the beginning [2].

2.1.2 Analytics and Ethics

The increase in collection of information and analytics has posed new requirements onprivacy and ethical issues. Lawyer Paul M. Schwartz [6] suggests a set of high-levelethical principles related to four di↵erent steps of the analytics process, namely collection,integration and analysis, decision-making, and review and revision. Schwartz composessome standards applying to all four steps that will be summarized below.

When using the analytical process companies should adhere to existing legal requirements,beyond the legal requirements companies need to assess whether they follow cultural andsocial norms considering acceptable behavior, they should also assess how their use ofanalytics will impact stakeholders’ trust in the company. Furthermore, companies shouldconsider both positive and negative consequences of their analytics and they should alsoimplement necessary security to protect the collected data [6].

The following is a segment taken from the terms of service of Google Analytics:

”You will not (and will not allow any third party to) use the Service to track,collect or upload any data that personally identifies an individual (such asa name, email address or billing information), or other data which can bereasonably linked to such information by Google. You will have and abide by anappropriate Privacy Policy and will comply with all applicable laws, policies,and regulations relating to the collection of information from Visitors. Youmust post a Privacy Policy and that Privacy Policy must provide notice ofYour use of cookies that are used to collect data. You must disclose the use

15

of Google Analytics, and how it collects and processes data.”

(§7 ”Privacy”, Google Analytics Terms of Service [7])

In addition to the company collecting user data, the user data will also be available toGoogle since it will be stored and processed on their servers. Google describes their use ofthe collected data in paragraph 6 in their terms of service. They specify three conditionswhen they may use the collected data, namely if the company using Google Analyticsgive their permission to Google; if it is required by law that Google uses it to protecttheir business, a third party, or the public; or if Google needs to hand out collected data,with restrictions, to third parties to carry out tasks on Google’s behalf [7].

Following is a summary of what people on the Internet have discussed on the matter.Since the following sources of information have not been published in scientific journalsbut on the web it should be read with an extra touch of criticism, however they are stillinteresting for the thesis work.

According to Frank Buytendijk the tool of analytics has a moral imprint [8]. Buytendijkhas more than twenty years of experience of software implementation consultancy, projectmanagement, as well as strategist and business executive roles [9]. He claims that manybusiness analysts consider their work to be merely discovering the truth and opportunity,and that it is up to their managers to decide what to do with it. If you do not act onthe data you are not doing bad things by collecting it. In continuation, he brings outthe fact that modern technology answers questions that we did not ask, which could betroublesome. Analytics is full of ethical issues, but not only concerning privacy issues. Itis sometimes hard to determine what data or knowledge is harmful. He concludes that itis necessary for organizations to think about ethical issues and discuss them, but also thatthere needs to be a public debate about it since ethical issues touch all stakeholders [8].

Lukas Oldenburg, senior digital analyst and team leader at Swiss-based e-business com-pany Unic [10], discusses his experiences working with Google Analytics and the issue ofethics related to it [11]. Oldenburg has finished the Web Analytics Association’s Awardof Achievement at the University of British Columbia [12]. He discusses what good mightcome from tracking cookies and collection user data. Next he tries to define what a goodjob is in terms of ethics. Thereafter he ponders in what ways digital analytics is goodand if it creates more than one-sided happiness. He summarizes four bullets that justifiesdigital analysis as follows: digital analysts make reaching goals easier, digital analyticshelps website companies and owners create value by reducing costs and increasing rev-enue, digital analytics creates value for Internet users by improving website usability andcontent as well as making them more customer-centric, and finally digital analytics cre-ates justice in organizations meaning that right position of the main content no longerdepends on the highest paid person’s opinion but on the A/B testing performed. His con-clusion is that similar to database administrators, website developers and other peopleworking with sensitive data should be responsible and reliable people [11].

Finally, the Digital Analytics Association defines the Web Analyst’s Code of Ethics [13]with the purpose of treating consumer data with respect and attention. Lamont Wood,a freelance writer, presented a similar code of ethics with the INFORMS Code of Ethicsfor Certified Analytics Professionals in an article where analytics professionals saw the

16

value of such a code [14]. The Web Analyst’s Code of Ethics is also referred to byOldenburg [11]. Refer to [13] and [14] to read the full codes of ethics.

2.2 Agile Software Development

Agile software development is a naming of software development methods that incorporatedevelopment in an iterative and incremental manner. They were compiled to developsystems with limited time spent on analysis and design and thus to develop systemsmore quickly [15]. The agile manifesto [16] reads as follows:

”Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan”

(The Manifesto for Agile Software Development, 2001)

Software development is performed through short feedback-loops to achieve a desirableand predictable outcome. Agile development depends on lightweight processes and agilitycould be defined to strip away the heaviness of traditional software development meth-odologies in order to accomplish quick responses to changes in environment and userrequirements as well as accelerated project deadlines to name a few. Agile methods di↵erfrom traditional methods in their fundamental assumptions, approach to control, man-agement style, knowledge management, role assignment in the teams and of the customer,project cycle, development model, and the organizational structure [17, Ch. 1]. Sections2.3 and 2.4 will further describe the agile methods or frameworks known as Scrum andKanban.

2.3 Scrum

Scrum is a framework for developing and sustaining complex products that has been usedsince the early 1990’s. The description of Scrum in this section is derived from foundersKen Schwaber and Je↵ Sutherland’s The Scrum Guide [18] and they define Scrum asfollows:

”A framework within which people can address complex adaptive problems,while productively and creatively delivering products of the highest possiblevalue.”

(Ken Schwaber and Je↵ Sutherland, The Scrum Guide, July 2013)

Scrum is based on empiricism which asserts that knowledge comes from experience anddecisions based on the current knowledge of the situation. To control risk and optimizepredictability Scrum uses an iterative and incremental approach as described in agile

17

software development above. Three important founding keys are transparency, inspectionand adaption.

Transparency implies the importance of the fact that everyone involved in the projectunderstands the ”language” used in the group. In other words parts of the process mustbe visible to them who are responsible for the outcome, they must share a common under-standing of what is seen. Inspection refers to the importance of a continuous procedureof inspecting the Scrum artifacts such as product backlog and sprint backlog to makesure progress towards the sprint goal is made. By this inspection manner undesirablevariances will be detected early. However, this inspection must not be so frequent thatit gets in the way of the work. Adaption means to take action upon the result of theinspection. The adjustments must be made as soon as possible in order to minimize theharm caused by the undesirable variability.

Scrum defines four meetings to limit the amount of meetings taking focus from the devel-opment: Sprint Planning, Daily Scrum, Sprint Review, and Sprint Retrospective whichwill be further described in Section 2.3.2.

2.3.1 The Scrum Team

The Scrum team consists of the product owner, the development team and a scrum master.The team structure of Scrum is designed to optimize flexibility, creativity, and productiv-ity. The key to success of a Scrum team is that it is self-organizing and cross-functional,meaning that it is not depending on other roles than the ones included in the team, theteam are the ones to decide how they want to work, and they have all the knowledgewithin the team needed to complete the project.

The purpose of the product owner is to maximize the value of the product and the workperformed by the development team. The product owner is accountable of the productbacklog and is the one who owns the product backlog. Managing the product backlogincludes clearly expressing items in the backlog, prioritizing the items to best achievegoals and missions, ensuring that the backlog is transparent, clear, and shows the workto be performed next, and to ensure that the development team understands the itemsin the backlog.

The developers in the team have no titles or roles and there are no sub teams in the group.They manage their own work and they are the only ones who create the increment. Theyare responsible for delivering a releasable increment by the end of the sprint.

The scrum master is responsible for the interaction between the team and the outsideworld to make sure that the team is able to do their work as e↵ectively as possible. Thescrum master makes sure that the team learns the Scrum process and keep to the rulesof Scrum. The scrum master services both the product owner and the development teamto make the Scrum process as fluent as possible.

18

2.3.2 Scrum Events

There are some events that define the Scrum process: the sprint, the sprint planning,the daily scrum, the sprint review, and the process of backlog refinement. A sprint is alimited period of time, at most 4 weeks and often shorter than that, during which theteam commits to a set of user stories presented in the product backlog. By the end ofthe sprint a usable and potentially releasable product is created. A new sprint startsimmediately after the end of the preceding sprint. A sprint goal is associated with eachsprint and during the sprint no changes are made that may endanger the sprint goal.Each sprint may be viewed as a project with no more than a one-month horizon. Thesprint planning is a time-limited meeting where the scrum team decides on what userstories to include in the sprint and what the sprint goal will be. The sprint planninganswers the following questions:

• What can be delivered from the upcoming sprint?

• How will the work needed to deliver the release be achieved?

Daily scrum is a fifteen minutes long meeting where all team members participate. Thedaily scrum helps the team create a plan for the following 24 hours. The daily scrumtakes place at the same time and place every day. The daily scrum is also commonlyreferred to as a stand-up since the team is supposed to stand up during the meeting tomake sure it does not take too long. During the stand-up each team member takes a fewminutes to answer the following questions:

• What did I do yesterday to help the team meet the sprint goal?

• What will I do today to help the team meet the sprint goal?

• What, if any, problems do I see that prevents me or the team from meeting thesprint goal?

The sprint review is a meeting held by the end of the sprint where the scrum team andstakeholders invited by the product owner, collaborate about what was done in the sprint.The development team shows the release that they have made in the sprint and discusseswhat went well, what problems arose, and how they were solved in the sprint. The resultof the sprint review is a revised product backlog. The sprint review is also commonlycalled sprint demo, since the result of the sprint is being demonstrated to stakeholders.

The sprint retrospective is a meeting held after the sprint demo, attended by the devel-opment team and the scrum master. During this meeting the team is able to discusswhat went well during the sprint, what they do not want to take with them to the nextsprint and what they want to bring into the next sprint. The topics discussed duringthis meeting is linked to people, relationships, tools, and the process. The team shouldthen create a plan for how they will implement the improvements identified during theretrospective in the next sprint.

Backlog refinement is the process of continuously improving the items in the productbacklog. If the developers experience that the items in the product backlog are notexplicit enough they can announce a meeting together with the product owner where

19

Figure 4: The Scrum process

they prepare for the next sprint by defining the next-coming items clearer so that everyteam member understands what it means. Another phrasing of backlog refinement isbacklog grooming.

2.3.3 Scrum Artifacts

Some artifacts that play a vital part of Scrum are the product backlog, the sprint backlog,and the definition-of-done. The product backlog is an ordered list that contains all therequirements of the product to be built and is managed by the product owner. Eachitem in the product backlog has a description, an order, an estimate and a value. Thesprint backlog is the set of items from that product backlog that the team chooses toinclude in the sprint together with a plan for completing the product of the sprint and toachieve the sprint goal. It is important that everyone involved in the project shares thesame definition-of-done to make it possible to describe an item in the product backlog as”done”. The definition-of-done is vital in order to maintain the necessary transparencydescribed earlier. Figure 4 visualizes the iterative manner of the Scrum process.

In addition to the artifacts described above it is also common practice to use a scrumboard to visualize the Scrum process and the progress of sprint backlog items throughoutthe sprint. The scrum board is divided into columns with each user story or backlogitem on a separate row. The tasks included in the backlog item are written on notes andmoved from column to column as it goes from not started to complete. An example of ascrum board is visualized in Figure 5a.

2.4 Kanban

Kanban in software development is an evolution from the original Kanban that wasintroduced in the late 1950’s in the Japanese manufacturing industry by Taiichi Ohno atToyota. Kanban is a Japanese word that could be translated into signboard in English andthe approach is used as a scheduling system for visualizing work in progress [19]. ThusKanban is a lean approach to agile software development. Lean software development

20

(a) A Scrum board (b) A Kanban board

Figure 5: Example boards

consists of seven principles: eliminate waste, amplify learning, make decisions as lateas possible, deliver as fast as possible, empower the team, build integrity, and see theentirety [20, 21].

The core of Kanban incorporates the following aspects:

• Visualize the work flow

• Limit work-in-progress (WIP)

• Measure the lead time

The work flow is visualized by splitting what is to be done into a backlog similar to theproduct backlog used in Scrum. In Kanban each piece is written on a card, called ticket,and put up on a wall. The wall can be either physical or digital. Named columns areused to illustrate where a ticket currently resides in the work flow.

The work-in-progress is limited by deciding for each column how many tickets are allowedto reside there at the same time. In this way bottlenecks and issues will be discoveredearly. For each ticket the lead time is measured, i.e. the time it takes for the ticket to gofrom not started to complete. The process is then optimized to minimize lead time andmake it as predictable as possible [22].

Figure 5b shows an example of a Kanban board visualizing the work process. The limitof the number of tickets in each state is denoted by the numbers above each column inthe work flow.

Studies show that the use of Kanban in software development results in many benefits.Improved lead time to deliver software, improved quality of software, improved com-munication and coordination of the project, increased consistency of delivery, and evendecreased customer reported issues [19].

21

2.5 Agile at Valtech AB

Most of the software development teams at Valtech use Scrum or a subset of Scrum intheir daily work. The Scrum process is often modified to suit the team better and thusScrum is not always practiced exactly as it is described above. It is common to removeand add meetings that are needed to better suit the work process for the respective teams.The extent to which Scrum is followed depends on factors such as how long the team hasbeen working together and how well do the team members know each other to name afew.

Some of the development teams at Valtech use Kanban in their daily work. Especially,Kanban is used in the live team at Valtech since the iterative nature of Scrum is notalways applicable to the tasks being performed there. The live team works with themaintenance and further development of projects that are formally completed, but thecustomer wants to extend or continue on a lower velocity.

In addition, many teams administrate their product backlog through online organizationtools such as Trello [23] and JIRA [24] since that greatly facilitates the process of updat-ing, prioritizing, and sharing items among team members and the client’s representatives.

2.6 Agile Analysis

Gary Angel wrote an article [25] on the topic of agile analysis where he discusses the dom-inance of agile software development and the di�culties with applying website analysisto it. Angel has more than twenty years of experience in decision support, customer rela-tionship management, and software development, and is responsible of the developmentof web analytics decision making tools at his company [26]. The article describes tradi-tional analytics as a waterfall process with conception, analysis, design, implementation,testing, and maintenance as the process steps.

He then discusses whether and how to adapt the analytics process to make it agile andstates that in doing so you will most likely keep to a design, implementation, and testcycle. The article also confirms the issue of tagging source code with measurements sincein an agile process, implementations may change from di↵erent sprints and this maya↵ect the tagging when code snippets are removed and variable names change. The taskof keeping track of the measurement requirements is also brought up. Should they beseparate from the user stories or should it be meta-data? He discusses this user storyissue further by talking about how the risk of measurement user stories may get droppedin prioritization meetings [25].

Himanshu Sharma has more than eight years of experience in search engine optimization,pay-per-click marketing techniques, and web analytics [27]. He wrote an article on theuse of agile analytics to quickly solve conversion problems [28]. According to Sharma,the aim is to conduct a very focused analysis by quickly deliver recommendations andimplementations to solve conversion problems. He finds that how quickly and often youcan create value for customer defines how agile you are.

22

Sharma continues to define a number of tips to adopt agile methodologies that will besummarized here. Firstly, focus on solving customer’s problems instead of KPIs whichbasically means to talk to the customer about the problem and thus quickly find thesolution. He also talks about being omnipresent, i.e. being present everywhere and thusbeing aware of what happens everywhere in the organization such as activities, events,and news. Moreover, he advices to keep records of all the changes that a↵ect your datain order to conduct precise analyses. He concludes with three last advices, namely tosolve problems along the way, to stop being a perfectionist, and to not conform to anyparticular process to find and fix problems.

2.7 Integration of UX

Ways to integrate User Experience (UX) in the agile software development process hasbeen analyzed and discussed in many research and case studies over the years [29]. Manystudies agree that integrating UX in the development process is a necessity to obtain asgood products as possible. They also agree that this integration comes with some issuesthat needs to be addressed in order to successfully make UX a part of the agile softwaredevelopment process [29, 30, 31]. The studies present di↵erent ideas on how to approachthese issues as will be briefly presented in this section.

Mona Singh presents a version of Scrum called U-Scrum [32] that extends traditionalScrum with usability in order to improve the user experience of the product. Singh iden-tifies limitations to traditional Scrum such as not teaching user feedback during sprintsand thus there are few possibilities in Scrum to build prototypes and validate require-ments before the implementation is started. Scrum is directed more towards completingdevelopment than enhancing user experience. In general, U-Scrum works by having twoproduct owners, one responsible for the usability and one traditional product owner.It should be noted that these two product owners are peers. Similar to other methodsU-Scrum uses user personas to perform user tests during the development process [31, 33].

Many of the studies performed on the subject identifies an approach using iterationswhere UX designers work one iteration ahead of the software development team in orderto complete as much as possible of the design before the implementation starts [29, 30,31, 32, 33].

23

3 Theory

3.1 The Lean Startup

The Lean Startup by Eric Ries [34] aims to provide entrepreneurs, in startups as well asin large companies, with a way to shorten product development cycles, measure actualprogress without reliance on vanity metrics, and help learning to know what customersreally want. Instead of wasting time creating elaborate business plans Ries teaches a wayto test visions continuously, and to adapt and adjust before it is too late [34, the cover].

The book is inspired by vanity metrics and lean thinking. Lean thinking incorporatesmaking use of the knowledge and creativity of the individual workers, to shrink batchsizes, to keep to just-in-time production as well as inventory control, and to acceleratethe cycle times [34, p. 18]. The following section is a brief review of the informationnecessary for this thesis work.

3.1.1 Validated Learning

Startups exist to learn how to build a sustainable business and that learning can bevalidated scientifically through performing frequent tests and experiments. This wayevery element of a vision can be tested. Validated learning is a unit of progress used inthe Lean Startup method. By using such scientific learning it is possible to discover andeliminate sources of waste in entrepreneurship [34, p. 8-9].

3.1.2 Minimum Viable Product

A Minimum Viable Product (MVP) is that version of the product that is the smallestversion possible to be able to make it through a full turn in the build-measure-learnfeedback loop, further described in Section 3.1.3. This means that it is the product thattakes the least amount of development time and e↵ort to make it through the loop. Tobuild an MVP requires extra time and work since it must be possible to measure itsimpact. In addition, it may also be necessary or convenient to show it, or even sell it,to the potential customers in order to measure their reactions to it [34, p.77]. MVPsrequire courage to test the assumptions that exist in a project. If the customers react inthe expected way in the test, the conclusion that the assumptions were correct could bedrawn [34, p. 109].

3.1.3 Build Measure Learn

The build-measure-learn feedback loop is a steering wheel that allows for constant ad-justments throughout the product development process, instead of making complex plansbased on numerous assumptions. With build-measure-learn it is possible to decide whetherit is time to pivot or if it is better to persevere at certain steps in the process. To pivotmeans to take a sharp turn and change strategy while persevere means to continue on

24

Figure 6: The Build-Measure-Learn feedback loop

the current path [34, p. 22], a method for how to determine this is further described inSection 3.1.4. The build-measure-learn feedback loop is visualized in Figure 6.

The goal is to minimize the total time through the loop. To be able to use the build-measure-learn method it is necessary to identify which hypotheses to test. The mostimportant hypotheses are the value hypothesis and the growth hypothesis which in leanstartup are called leap-of-faith assumptions. When these hypotheses are defined the buildphase of the build-measure-learn feedback loop is entered with an MVP. When the MVPis built it is time to enter the measurement phase. The most important part of themeasurement phase is to determine if the development is leading to progress or if thee↵orts are a waste of time. If the team is building something that nobody wants it isirrelevant whether it is done on time and budget. The method described to measure thisis called innovation accounting which is a quantitative method that allows the creationof learning milestones which is an alternative to the traditional business and productmilestones. The learning milestones are useful to entrepreneurs to measure the progressand for managers to assess it [34, p. 76-77]. After completing the build-measure-learnfeedback loop it is time to decide whether to pivot or persevere.

3.1.4 Pivot or Persevere

To decide whether to pivot from the original strategy or to persevere is one of the hardestdecisions to take for an entrepreneur. If any of the hypotheses put up before enteringthe build-measure-learn feedback loop fails after the loop, it is important to come upwith another strategic hypothesis. The lean startup method allows for entrepreneurs andstartups to pivot sooner and thereby build capital-e�cient companies [34, p. 77-78]. Thepivot is a structured course correction designed to test a new fundamental hypothesisabout the product. Companies that do not come to the conclusion to pivot when it isnecessary may land up in a ”land of the living dead”, where they are neither making

25

progress nor perishing. Successful pivots make path for sustainable business. The goal ofthe learning milestones of the build-measure-learn loop is for it to provide enough data tomake the decision of pivot or persevere easier to make. A pivot requires to ”keep one footrooted” in what is already learned while making the fundamental change of the course inorder to achieve greater validated learning [34, p. 149-154].

The symptoms of a necessary pivot are the decreasing e↵ectiveness of product experimentsand the feeling that product development should be more productive. Ries suggests thata recurring pivot-or-persevere meeting could be scheduled in order to make this decisioneasier to make and avoid postponing it. It is necessary that both the product developmentteam and the business leadership team are present at this meeting. Necessary reports tobring are the result of the optimizations done by the development team and a comparisonof how those results appear against expectations over time, and the business leadershipshould bring detailed reports on their dialog with customers [34, p. 164].

3.1.5 Lean Startup Applied to Mobile Application Development

UX veteran Beverley May applied the Lean Startup philosophy to a smart phone ap-plication development project for dating [35]. Their experiences and lessons learned wereconcluded in five key lessons: to be aware of advice from others, to optimize the team andonly include really talented people that can be trusted to deliver, beware of technologicaldecisions that may cause problems ahead, make sure to perform thorough UX designwithout cutting corners to save time and money, and test everything over and over again.

They did not always keep to the agile framework as they had intended in the beginningdue to that fact that it was not always applicable to their situation. Most of the timethey only had one developer on the team and therefor artifacts of Scrum and similarmethodologies did not suit their work. However, it is stated that the introduction ofdaily scrums see section 2.3.1, really helped the team. May considers the introductionof daily scrums to be the most valuable process to help manage and monitor the teamperformance.

A conclusion is that it is very hard to create a startup even when following the principlesof the Lean Startup. At the time of writing, the team found themselves in the samesituation of having a working and nice looking product but no customers that Ries triesto avoid through the Lean Startup [34].

3.2 Lean UX

Lean UX by Je↵ Gothelf with Josef Seiden [36] is an application of the Lean Startup philo-sophy to perform UX design in agile environments. The three foundations of Lean UX aredesign thinking, agile software development, and the Lean Startup method. Design think-ing is a critical method for teams to collaborate across roles and consider product designfrom a holistic point of view. The agile software development is important since softwaredevelopers have been working in agile environments for a long time to continuously de-liver customer value in short cycles. The core artifacts of Lean Startup founding Lean

26

UX are the build-measure-learn feedback loop and the MVPs and thus also hypothesisthinking [36, p. 5-7].

3.2.1 Principles of Lean UX

To compose Lean UX, Gothelf defines a set of principles containing small and dedicatedcross-functional teams that are problem focused. Furthermore, progress should be meas-ured by outcomes and not output, waste should be removed, and work should be donein small batch sizes. In addition, continuous discovery is important as well as ”gettingout of the building” to meet potential users, and to have a shared understanding andcollective knowledge within the team. ”Team rock stars” and gurus should be avoidedsince elite experts may break down the team and reduce collaboration, externalizing thework to a public view like white boards and artifact walls, and making over analysis interms of asking customers in the field. Another important principle is the learning overgrowth since it is hard to build the right thing from the beginning, and also to create anenvironment that permits failure to find the best solutions through experimentation andlearning. Lastly, refocus the design process away from the documents it creates to theoutcomes it achieves to get out of the deliverable business [36, p. 7-12].

3.2.2 The Hypothesis and MVP

According to Gothelf, hypothesis creation is the main tool to achieve outcome-focusedwork. It is the starting point of the project and states a clear vision for the work. Basicallyit is about expressing assumptions in a testable form [36, p. 17]. The hypothesis formatis presented as follows [36, p. 22]:

”We believe [this statement is true].We will know we are [right/wrong] when we see the following feedback fromthe market:[qualitative feedback] and/or [quantitative feedback] and/or [key performanceindicator change]”

Using this format it is possible to create sub hypotheses on the format: ”We will [createthis feature] for [this persona] in order to achieve [this outcome]” [36, p. 30]. By prior-itizing the hypotheses a set of explorable paths are identified. Those are tested throughthe MVP.

Among other things, search logs, site usage analytics, and A/B testing are tactics pro-posed to get customer feedback. Funnel analysis is highlighted in site analysis to seewhere customers are dropping o↵ [36, p. 88]. Gothelf mention some development en-vironments that for di↵erent reasons does not allow frequent releases, such as packagedsoftware, embedded software, or a development environment where continuous deploy-ment is di�cult or impossible. In this case Lean UX may not be a great fit for the team,because too much time and energy will be spent on getting the necessary market feedbackto make these techniques work [36, p. 98].

27

3.2.3 Suggested Organization Shifts

Gothelf concludes by suggesting a set of necessary organization shifts that may be neces-sary to be able to apply Lean UX. Firstly, shifting focus on output to outcomes, movingfrom limited roles to collaborative capabilities where all roles, project managers, softwaredevelopers, UX designers, quality assurance people etc., work together, as well as em-bracing new UX skills to obtain team owned product design and a leadership in designsessions. Creation of small cross-functional teams that do not rely on individual heroesto achieve the goal, placing speed and not aesthetics first, as well as valuing problemsolving.

Furthermore, shifting agency cultures may be necessary, customers pay for deliverablesand not outcomes and also work to create a contractual relationship with third partyvendors working with the organization. The UX dept requires the team to commit tocontinuous improvement of user experience. Moreover, work around requirements ondocumentation standards by designing and testing first and document afterwards. It isalso necessary to be realistic about the environment since a change such as Lean UX isbig, thus approach these concepts by testing small parts and show the organization thegained achievements. Lastly, by using Lean UX to step away from the ordinary road mapused by organizations to coordinate activities it becomes very important to communicatethe planned activities of the team to the concerned portions of the organization [36,p. 109-120].

3.3 Business Impact

Business impact or impact goal, is the impression left by a project when it is com-pleted [37, p. 231-232]. Because of this, the impact goal is the most important goal of allthe goals tied to a project. The impact goal should be known and formulated before theproject is initiated since it is the only reason to why the project is started. To start aproject without a specified impact goal is the same as starting a large work task withoutbeing able to explain the reason why it is done. The impact goal contributes to thefoundation necessary to be able to measure each and every action within the project [38].

3.3.1 Business Impact Management

Business impact management is founded on the notion that the success of the projectarises in the use of the product developed through the project, therefore everyone shouldsteer towards the expected business impact when managing their project. The singleaim of business impact management is to help the project steer towards those impacts.In order to do this, business impact management maintains some roles with explicitresponsibility. In other words, there is a process to navigate against the impact goals.

Business impact management means to proceed from the desired impacts created throughthe use of methods such as business impact mapping and analysis of the target group.Thereafter, solutions with explicit connection to the desired impacts are designed such

28

that both the client and the supplier maintains the same image of what should be done.This could be done through interaction design.

Subsequently, user tests with realistic tasks are performed in a realistic environment.The system is then adjusted by using the result of the user tests as an answer. Businessimpact management can be used together with any system development method. Thedi↵erence by using it is that activities can be performed with increased precision and theproduct development process as a whole becomes much more e↵ective [39]. The use ofimpact mapping will be further described in Section 3.3.2.

3.3.2 Business Impact Mapping

Impact mapping is a technique for planning projects that prevents organizations fromgetting lost while developing new products. Impact mapping makes use of clear commu-nication assumptions which helps alignment between development teams’ activities andbusiness objectives. Development teams get a road map and the business sponsors get abig-picture view, thus better road map decisions are achieved. An impact map visualizesthe dynamic relationship between delivery plans and their environment. Furthermore, itencapsulates the most significant assumptions and the delivery scope.

The use of impact maps helps adapting plans e↵ectively and reacting to changeovers.By putting deliverables in an environment of impacts the right outcomes are achieved.Impact mapping communicates underlaying assumptions and allows for teams to testthem [40]. Impact mapping emerges from asking four sets of questions visible in Table 1.

Why... ... are we doing this?Who... ... can produce the desired e↵ect?

... can obstruct it?

... are the consumers or users of our product?

... will be impacted by it?How... ... should our actors’ behavior change?

... can they help us to achieve the goal?

... can they obstruct or prevent us from succeeding?What... ... can we do as an organization or a delivery team, to

support the required impacts?

Table 1: Business impact mapping questions

An impact map puts all the deliverables in the context of the impacts they are supposedto support as well as throwing away deliverables that do not contribute to any criticalimpact for a specific goal [40]. An impact map shows the e↵ects of technical deliverablesfrom a business perspective. With help of the impact map the organization responsible forthe delivery maintains its focus, and are able to prioritize and define actions to improvethe quality of the delivery.

Once a delivery is released it is possible to measure the change in users’ behavior andthe impact of the overall objective. This process can then be iterated in order to decide

29

whether to continue on the same part of the map or change strategy to another part ofthe map. By using methods such as impact mapping where deliverables are mapped toa business goal the number of business goals and impacts worked on at the same time islimited. This e↵ect is similar to the one achieved by lean development processes wherethe work in progress is limited [40, Ch.2].

3.4 Google Analytics

Google Analytics is a tool that makes it possible to track user behavior on websites aswell as mobile applications. The following is a subset of all the tra�c and data that canbe tracked, in both real time and over time [41]:

• The number of visitors

• The geographical spread of the visitors

• What pages on the website that visitors interact with

• What goal conversions that took place

• Which pages are end pages, i.e. which page is the last before leaving

3.4.1 Implementation

To use Google Analytics on a website it is necessary to insert small fragments of code inthe pages of the website. The fragments will activate the tracking and send data to theuser account. From the user account it is possible to view the data through diagramsand plots. From the account interface it is also possible to define goal conversions andconfigure tracking of tra�c. It is possible to insert the code with JavaScript and sinceGoogle Analytics uses namespacing for its JavaScript there is no risk of overriding codethrough conflicting variable or method names [42].

Figure 7 shows an example of a JavaScript code snippet used to track a web page. Forthis example code to work the UA-XXXXX-X string at line 7 needs to be replaced withan actual web property ID and the code snippet should be pasted before the </head>tag in the website template [43]. Apart from the missing property ID, the code snippetin Figure 7 is used in the main HTML of the new website being developed for Valtech.Google Analytics was recently upgraded to Universal Analytics, which means that theJavaScript API changed from ga.js to analytics.js [44].

3.4.2 Dashboards

Dashboards display a summary of the di↵erent reports of an account. The summariespresent an overview of the performance of properties chosen for tracking. In the dashboardit is possible to view many metrics at the same time and thus compare and see correlationsbetween reports [45].

30

Figure 7: A Universal Analytics code fragment

3.4.3 Custom Reports

It is possible to create custom reports in Google Analytics containing dimensions andmetrics. Dimensions are ways to describe users, sessions, pages, products and events, forexample a city or a browser. Metrics are numeric measurements such as page views andbounce rate. By creating custom reports it is possible to decide how the data should bedisplayed in the report [46].

3.5 Theory and Research Questions Connected

Section 1.2 presented the research questions of the thesis work: ”How should measure-ments be used between sprints?”, ”How should the agile team be composed?”, and ”Howcould early release to production be achieved?”. These research questions were composedto answer the main question: ”How can the work of collecting and analyzing quantitativeuser data from websites be integrated into the agile work process at Valtech?”.

The Lean Startup and Lean UX present guidelines for how other businesses have suc-ceeded in making their organizations guided by learning and statistical examination in anagile environment. The theory presented in those books composes a foundation for howto achieve a usable and successful product in every increment, thinking about Scrum butalso agile in general, and thus also material to figure out how to make projects releasablebefore the product is finished. Lean UX brings up some necessary organization shiftsthat connect with the second question concerning team composition. Business impact iscentral to this thesis work and connects mainly to the first question concerning measure-ments between sprints as well as promoting the third question about early release andthe similar need for continuous deployment.

31

4 Methods

4.1 Literature Study

The thesis work was initiated with a literature study to gather knowledge and under-standing of current related work and information necessary to know. Scientific articlesfrom well-known databases have been gathered as well as relevant publications on theweb, such as blog posts and other interesting texts. When reading web publications thatare not published in scientific journals it is important to keep in mind that these textsshould be read with an extra thought of criticism. Therefore, blog posts and similar textsused in this thesis work can only give a hint of what is being said and thought of on thesubject.

4.2 Google Analytics Implementation

In order to get a better understanding of website analysis and Google Analytics I spentsome time on implementing scripts to collect user data on an internal project at Valtechand taking part in their Scrum process. The website experimented with was underconstruction and thus the scripts were implemented and set up towards a Google Analyticsaccount tied to the test server of the project.

4.3 Observation of an Agile Software Development Team

The thesis work was performed in the same location as a software development teamand thus made it possible to observe them in their daily work. Stand-up and sprintplanning meetings of two di↵erent teams were attended in order to get insight into theirwork process and thoughts on were in the process it would be possible to perform theintegration proposed by this thesis work.

4.4 Pre-study Interviews

4.4.1 Interview Setup

During the pre-study phase a total of nine interviews were conducted. The first twointerviews were conducted in order to get a general understanding of how Valtech worksin terms of agile methods and user tests to identify what is already established at thecompany. These two persons will be labeled ”general information interviewees”. Pro-ceeding from the general information sessions, the remaining interviews were prepared.Apart from the ninth interview, the interviews were conducted with people from threegroups identified as interesting for the thesis work:

• Project managers,

• Web analysts, and

32

• Software developers

Two people representing each group were selected for an interview. These interviewswere performed in order to gain understanding of their respective view of integrating thework of measuring and analyzing website data in the agile work process. Furthermore,the aim was to get ideas on how to define such a work process from talking to relevantpeople. The overall aim was to identify di�culties, needs, and opinions of each categoryof people.

The interviews lasted for between 20 to 40 minutes. Every interviewee was asked forpermission to record the interview and since none of them objected to this all the inter-views were recorded. Furthermore, none of the interviewees objected against the namingof their true names in the report, however it was decided to keep the interviewees an-onymous since the impression was that their names would not further the purpose of thisthesis work. The interviews were conducted in Swedish since both the interviewer andall the interviewees have Swedish as their native language. Since this report is written inEnglish, all answers or part of answers included in this report have been translated intoEnglish.

4.4.2 The Interviewees

The following section specifies the interviewees selected for the study. The intervieweesare numbered according to the order their respective interview took place. The first twointerviewees consisted of one UX designer and a consultant manager. The ninth personwas interviewed since that person was said to have opinions about the subject of webanalysis and experience of working with it in their projects.

General information interviewees:

Interviewee 1 has a background as system developer with a Master’s degreeof Computer and Systems Sciences specializing in interactive systems. Inter-viewee 1 works as a UX strategist with experience in pre-studies, researchand investigations, strategic work tasks and similar issues that need to beaddressed before the project starts and also in the beginning of the projectwork.

Interviewee 2 has worked at Valtech for the last ten years and is an exper-ienced developer in .Net, web development, system integration, and systemarchitecture. The person has also worked as tech lead and has recently tran-scended to consultant manager. The person has a Master of Science of En-gineering in Computer Technology.

Project manager interviewees:

Interviewee 3 has worked at Valtech for eight years as an agile project man-ager, web strategist, support for clients and product owners, agile coach, andanalyst. The person has a Master’s degree of cognitive science.

33

Interviewee 4 is a project manager and team leader that has worked mostlywith the same project during their whole time at Valtech. The person hasworked with quality management both at a company level and project level.The person is a university engineer in electronics with one year specializationof system development.

Web analyst interviewees:

Interviewee 5 has knowledge of both front end development as well as userexperience and works with web analytics tools. The person has a Master ofScience in Engineering in Information Technology.

Interviewee 6 works with digital strategy, e↵ect optimization, and marketingthrough the web. The person pushes for the lean startup philosophy andlean UX as ways to work. The person has a Bachelor’s degree in BusinessEconomics specializing in Marketing and has experience of web strategy andanalytics.

Software developer interviewees:

Interviewee 7 has experience of working as a consultant for 10 years andspecializes in mobile application development and web development. Theperson has a Master of Science in Engineering in Computer Technology.

Interviewee 8 is a back end developer specialized in Java and .Net and hasworked as a system developer for 14 years. The person is a university engineerin computer technology.

Art director interviewee:

Interviewee 9 works as an art director and perform design tasks related tomarketing, mobile application development, and web development. This per-son was interviewed as a team member with experience of analysis in theirteams.

4.4.3 Interview Questions

The general information interview questions focused on the agile work at Valtch todayas well as how Valtech works with user tests. The questions for the project managerswere divided into the subjects of project pre-study, project startup, business impact,integration of analytics, and agile methods.

The questions for the web analysts were centered around statistical and measurementdriven design and the Lean Startup philosophy described in Section 3.1. The questionsfor the developers were similar to the ones posed to the web analysts although they wereangled to suit the developers and aimed to identify opinions and possible di�culties.

34

There were also questions regarding what kind of people or team composition are neces-sary for the integration which were posed to all the interviewees. The full list of questionsin Swedish is included in Appendix A. All questions were phrased in an open manner inorder to increase the talk of the interviewee and get the most out of the interview. Ques-tions with answers such as ”yes” or ”no” were avoided to the furthest extent, howeverthat kind of questions were sometimes posed as a follow-up or a check-up question.

4.5 Evaluation of Proposal

The evaluation of the process proposed in Section 6 was carried out through three parts,namely a questionnaire, succeeding interviews, and an adaption of the product backlogin terms of identifying and rewriting user stories relevant for measurements. In this caserelevant for measurements means that the user story is coupled to the user interface ofthe product being developed. As described in Section 4.5.1 a set of four project teamsfrom Valtech were selected to evaluate the process proposal.

4.5.1 Participating Projects

The project teams selected for the evaluation are shortly described below. The projectsare numbered according to the order the respective evaluation session occurred. Twointerviewees were selected from each team as further described in section 4.5.3.

Project 1 is an e-commerce website that is currently live and being furtherdeveloped by a team at Valtech. The team is geographically spread such thatthree of the developers are located in Stockholm, one developer is locatedin Gothenburg, and the product owner, i.e. the client, is located in Skovde,Sweden. Both interviewees chosen from this project were back end developers.

Project 2 is a set of several projects for the same customer, a travel agency.The interviewees chosen from this customer were one back end and frontend developer and one UX person. The team is composed of both Valtechconsultants as well as other consultants.

Project 3 is a customer in the media industry providing several services relatedto television. The interviewees selected from this project were two developers,both back end and front end, whereof one was also working as a product ownerand thus was directly employed by the company and not a Valtech consultant.

Project 4 is a telephone network company where Valtech consultants aremostly focusing on UX work and the development is outsourced to anothercompany abroad. The project actually consists of several projects since thecustomer has ordered several di↵erent services, however the Valtech team isstill working together in subset constellations. The services being developedare related to both e-commerce and di↵erent types of mobile applications.

35

4.5.2 Questionnaire Regarding Project Measurability

Four parameters were identified that indicate to what extent a project is measurable.The parameters are countered as follows:

• Driven by the backlog vs Driven by benefit

• Customer satisfaction vs Customer benefit

The questionnaire was written in two versions, one profiled against the team members,i.e. mostly people from Valtech, and one version for the clients. Swedish versions of thequestionnaires are included in Appendix B.1 and B.2 respectively.

4.5.3 Interviews

The interview sessions were conducted with two people at a time. This approach waschosen to create a discussion and increase the amount of feedback possible to receivefrom the interviewees. Some general questions were also prepared, however the focus wasnot to ask those questions but rather to have the interviewees talk freely and reflect onthe process proposal. The interview questions prepared for the sessions are available inSwedish in Appendix B.3.

4.5.4 Inspection and Adaption of Product Backlog

The original thought was to let the interviewees of every project inspect a subset or theirrespective product backlog to rewrite the backlog items as an hypothesis as describedin Section 6.3. However, this scheme was realized with the project 1 only. The decisionto skip this element from the succeeding interviews was made for a number of reasons.Firstly, translating user stories to hypotheses was perceived as strained by the inter-viewees. Secondly, when writing hypotheses it is appropriate to start from the top andfurther detail the formulation as the work goes on, instead of working from the bottomwith detailed user stories to the top. Lastly, not all projects had a backlog to share, suchas project 2 and project 4, and along with the previous reasons the decision was made toskip the backlog exercise from those interviews. As will be discussed further the conceptof writing hypothesis was still discussed thoroughly during all sessions.

36

5 Pre-study Interview Summary

The following headlines summarize the interview answers into suitable groups. It shouldbe noted that the interviews were conducted with representatives from each identifiedgroup of interest. Thus the interviewees will highlight a subset of all the opinions andinformation possessed at Valtech and therefore it is possible that other people would haveanswered the questions di↵erently.

5.1 User Tests

The first general information interview gave an understanding of how Valtech works withuser tests in their projects. There are no defined templates to follow and how the usertests are performed depends very much on the context. Whether a project is in a startupphase, in a pre-study phase, or even in the implementation phase matters since thatwould call for di↵erent actions such as target group analysis and concept outline to namea few.

Common for all user testing is to spend time with a potential user to observe the personwhile he or she is performing a designated task during a limited amount of time. Whilethe test person performs the actions the tester will ask questions focusing on why theperson is doing this way, more than what the person is actually doing. The questionsregard how the test person felt while doing something, why the person did it that way,and so on. In the beginning of a project and possibly also in the pre-study phase it iscommon to use simple paper prototypes as well as schematic outlines.

During the project, continuous user tests are performed. Sometimes test persons arehired from an outside pool of test persons. The information gathered from the user testsis then integrated in the project by being part of the planning process of the projectwork. If the information is very specific and points to some important feature of theproject it will go directly into the product backlog as a new item. The information willalso influence the work of prioritizing the product backlog items before the developmentof new parts of the system begins. It is common for the UX personnel to work some timeahead of the rest of the team in order to prepare parts of the solution that will soon beimplemented.

5.2 Business Impact

Business impact was mostly recognized by the project managers interviewees and thepre-study interviewees. The web analysts did to some extent have experience of it. Thedevelopers did not get as many questions about business impact as the other interviewgroups did since the focus of the developer group was to learn their experiences fromworking with measuring user data and performing analysis.

The interviewees that had experience of working with or knew of business impact agreedthat it is something that is not worked on to the extent that it should. Impact goals are

37

almost always defined in all the projects that the interviewees had experience of, howeverthey were not always followed up by the end of the project as they should according totheir purpose of measuring the impact of a project when it is completed.

The interviewees agreed that this is something that should definitely be part of the projector at least it should be planned for someone to perform a follow-up on the product somespecific time after it has been launched. As interviewee 1 stated:

”... they usually say that it’s only when the product is launched that thejourney really begins, that’s when it’s so important to have a good organizationworking with content and work, and further development...”

Several of the interviewees gave an impression of uncertainty regarding business impactand impact goals. Even though they are defined it may be troublesome to figure outhow to measure them. Some impact goals may also be poorly defined which will furtherobstruct the process of following them up. Interviewee 3 stated the di�culty of impactgoals as follows:

”... they usually talk about S.M.A.R.T. goals, that goals should be measurable,timed, right in time and it should be realistic and all that. Often measurableis di�cult in complex businesses, to define goals that are truly measurable!”

The overall impression from the interviews was that business impact goals are very im-portant and must be worked on in all projects. Furthermore, according to interviewee 6it is impossible to drive design by statistics if business impact is not prioritized. However,the impression is that business impact mapping, i.e. to create business impact maps, isnot widely spread among the interviewees.

5.3 Design Driven by Statistics and Learning

During the interviews, questions were posed whether measurements of user behavior wereperformed in the development process. According to the interviewees some measurementswere actually performed and many of the projects they had worked on did have someamount of tracking.

According to interviewee 5 statistically driven design is all about discovering trends andusing knowledge from previous projects to make decisions in the current project. Onquestions regarding the lean startup philosophy one of the web analysts, interviewee 6,experienced the following:

”... I feel that many see this way to work as a threat to their needs, that itbecomes so very little volume in their assignments if they should just sit andcreate hypotheses and you don’t have time to think out a lot of nice interactionor customer travel maps to know all touch points and so on, it’s a much moreexperimentation driven and agile way of working, and it’s not everybody wholikes that...”

Another prerequisite brought up by the same interviewee is the cultural environmentnecessary to fully implement a statistically driven design and implementation process.

38

It is vital for the cultural environment to accept failure and that failed experimentsare part of the learning process. Furthermore, design driven by learning is necessarysince the rate of change is so large in the IT industry. Thus to learn and adapt tochange is vital to be able to compete with rivals and in an organization with shortfeedback loops where it is relatively cheap to perform experiments the organization hasa vast competitive advantage. Interviewee 3 highlighted the importance of performing arequirements analysis with the client regarding their need of measurements as follows:

”... what reports do you want, which segments are there, what are the meas-urement points, which conversions are there, are external systems going to betracked, i.e. the conversions that go from the site to some other external site,and so on. To do the homework and do a thorough requirements analysis oftheir measurement needs related to business impact goals is very important.”

It was also discussed whether it would be reasonable to add the measurement as somekind of acceptance criterion on user story level in a similar manner as software tests andcode review that are often part of such criteria. Several interviewees from the projectmanagers, web analysts, and developers, discussed di↵erent kinds of projects such asinformation websites opposed to e-commerce websites in terms of how measurable theyare. E-commerce sites have explicit impact goals, micro conversions as well as macro con-versions, while information websites may have ambiguous impact goals such as ”increasevisitors’ knowledge” on a certain matter.

In other words, to what extent the di↵erent projects work with data collection and analysisalso depends on Valtech’s clients. At the moment a large portion of the clients do nothave the web as their main activity and thus their websites may be a less importantchannel in their business. According to one of the interviewees, agile website analysisis easier to perform on maintenance projects since real time data is available and newfunctionality can be assessed immediately through continuous deployment.

5.4 Team Composition

Many of the interviewees suggested that a UX designer should be the one responsible ofthe measurement, collection, and analysis of the quantitative data. At least they mentionsuch a person suitable for that role in the team, as an answer to the question of what teamcomposition or modification would be necessary for the process to function. Interviewee6, one of the web analysts, states the following:

”... they have to know some interface development, at least to be able to talk todevelopers on the team, you should be a UX person interested in the interfacefor it to be truly e↵ective...”

Several of the interviewees discussed the need of some person with experience of meas-urements and analysis and it was discussed whether this person should be part of theteam during the whole project or if it is more suitable for that person to come by whenneeded. In order to perform the requirements analysis mentioned in Section 5.3, a personneeds experience to be able to ask the correct questions. The interviewees agree that thepeople working with analysis need to possess several competences, meaning they should

39

also know about for example front end or back end development in addition to workingwith analysis. If they possess multi-competence of this kind they are able to remain onthe team during the whole project, otherwise it is hard to motivate why they shouldremain on the team on full time.

Interviewee 5 identified some needs necessary to work with website analysis at Valtechtoday. Firstly, the number of people working with analysis and actively working withGoogle Analytics needs to increase. This was also agreed upon by the other web analystinterviewee 6. Furthermore, website analysis needs to be a bigger part of the projectbudget. According to interviewee 5 the people working with analysis today need to taketime from their other projects to work with analysis on several other projects as well,which is not always appreciated by their teams.

Both of the web analysts and also one of the project managers talk about the necessityof writing user stories and tasks that apply to measurements, either to write them ashypotheses instead of user stories or simply to include how it should be measured in thedescription. This necessity requires the team to be accessorial and collaborative regardingthe measurements in terms of both planning and implementation.

40

6 The Proposed Process

Agile system development is about creating products in an e�cient manner with regularcontact with client representatives and stakeholders as well as delivering a working incre-ment by each sprint. In order to perform website analysis in an agile manner data needto be gathered as soon as possible, processed and analyzed between sprints, and actedupon as soon as decisions are possible to make.

There are roughly three kinds of projects identified by this study: the projects whereno release has yet been done, the projects where the release process is started but theproduct is not yet finished, i.e. the work is still in progress, and the projects where theproduct is finished and may enter a maintenance phase. Di↵erent steps or parts of theprocess will be more or less suitable depending on what kind or phase a specific projectappertain to. Thus the aim is to provide a guide that is adaptable to the project at hand.

The following sections are a collection of what steps need to be taken in order to integratemeasurements of quantitative user data successfully into the project work at Valtech. Theguide presupposes an agile work process similar to the one described in Section2.2. Thesteps are numbered for reference reasons and thus they do not necessarily have to beexecuted in the order they appear. A (*) denotes a step in the proposed process that hasbeen modified after the evaluation, see Section 7.3. The project startup phase and theongoing project phase have also got some new bullets. The complete process after theevaluation is available in Appendix C.

6.1 Project Pre-study

It is very important that the business impact of the project is well defined as a first stepin the pre-study phase. Furthermore, a good way to define the business impact and makesure that everybody has agreed upon the purpose of the project is to perform workshopswith the client and representatives from Valtech with knowledge and experience of webstrategy. The workshops will also help define KPIs that are necessary for the measurementprocess. Thus the following steps will be useful:

1. Define business impact goals

2. Perform workshops with representatives from the client and Valtech

(a) Discuss and define KPIs

(b) Determine what conversions to track

3. Perform the questionnaire for evaluating the project measurability described inSection 4.5.2

41

6.2 Project Startup - Sprint 0

The startup phase is the stage where time is spent on preparing the project. Necessaryequipment is put in place, team building activities take place in order to startup theteam, test environment as well as production environment necessary for the softwaredevelopment are put up. An initial product backlog is compiled. In this stage it isimportant that the following steps are considered:

4. Necessary KPIs should be defined if they are not already set

5. Create a Google Analytics account for the project (*)

6. Set up the Google Analytics account in terms of

(a) Dashboards

(b) Custom reports

7. Show the client how to use the Google Analytics account to follow-up data

8. In connection with set up of test and product environments, make sure that theminimal tracking code of Google Analytics is added as soon as the initial websiteis put up so that data will be gathered from the beginning and tracking will beperformed in every environment

9. Put up a screen showing a Google Analytics dashboard of the three most importantmacro conversions based on the business impact goals identified in the pre-studyphase

10. Make sure that relevant product backlog items delivered during this sprint 0 arewritten as hypotheses (more about hypotheses in Section 6.3)

6.3 During Ongoing Project - Sprint 1 to Sprint N

During the project work the following steps need to be thought of. Relevant user storiesrefer to product backlog items that are related to graphical user interfaces and thusrelevant in order to collect user data and measure user behavior. The steps definedfor this project phase are supposed to be performed in the iterative manner of agilemethodologies as visualized in Figure 8:

11. Ensure that the dashboard configured in sprint 0 functions as intended, in otherwords make sure that the intended measurements are implemented. This could bedone by using special software that animates click through the website, such asSelenium tests and PhantomJS

12. Product backlog prioritizing:

(a) Use the previous measurements from this project to prioritize the backlog

(b) Use knowledge and experience in terms of measurements from previous pro-jects

42

13. Backlog grooming - Think about analysis and how to measure user behavior foreach backlog item discussed during the grooming (*)

14. Sprint planning - Relevant user stories should be written as hypotheses: (*)

”We believe that [this capacity] will result in [this outcome].We will know we have succeeded when [we see a measurable signal]”

15. Build MVPs from the hypotheses above

16. Implement necessary measurements during implementation of the new functionality

17. Aim for continuous release as early as possible. In other words make sure to releaseto production after each sprint. Necessary sub steps for this is to set up a routinefor

(a) Continuous integration,

(b) Automated tests, and

(c) Continuous deployment

18. Sprint demo - Include measurements from the sprint to increase awareness of ana-lytics (*)

Figure 8: The Scrum process with additional process steps

6.4 Project Completion

In this stage there are two possible ways for the project to take, the first one being ValtechLive and the other being a maintenance organization not tied to Valtech, for example aninternal maintenance organization of the client. If the project ends up at Valtech Livethe following steps are necessary:

19. Make sure to create a strategy for further analysis

20. If further development is performed:

43

(a) Continue to release after every sprint (*)

(b) Add measurements to relevant new functionality, in consultation with the cli-ent or according to the planned strategy

21. Follow-up measurements on a regular basis, as a suggestion once every month

If the project does not end up with Valtech, make sure that there is a plan for how towork with continuous measurements in order to follow-up on the business impact goalsdefined in the project pre-study.

44

7 Evaluation

7.1 Questionnaire Results

The following section summarizes the answers and comments on the questionnaires, sep-arated in team and customer perspective. The questionnaires are available in Swedish inAppendix B.

7.1.1 Team perspective

A total of fourteen team members answered the questionnaire. The division of the teammembers main function on the team were as follows: five UX designers, two art direct-ors, six back end developers, and three front end developers, whereof two team membersidentified themselves as both front end and back end developers. 79% of the respond-ents considered ”the project to be a good investment for the customer” as the mostimportant parameter of a project and 21% considered ”the project fulfills the requestedfunctionality” as the most important parameter.

The respondents provided several comments on this question where the overall opinionwas that all parameters are important, however some of them were more important.Further opinions were that the customer has a responsibility to choose whether time,budget, or quality is most important, it should be meaningful to work with the projectas a consultant, that the customer gets value for their money, and as a consultant youoften have to compromise between time, budget, and quality.

The average of the respondents’ ratings of the importance of all the project parameters isdisplayed in Table 2. In the comments on this question it was concluded that the use ofthe product is more important than goals and constraints, time and budget. Furthermore,it was stated that in the digital world of today, value is everything. Therefore having acustomer focusing too much on time or budget could result in incomplete services beingreleased on the market and could in continuation harm the customer’s trademark.

Parameter Rating (out of 5)The project is completed on time 3.3The project is completed within budget 3.5The project fulfills the function required 4The project is a good investment for the customer 5

Table 2: Importance of parameters

The importance of a clear backlog was an average of 3.8 out of 5. The respondentscommented that it is important that the backlog is alive and flexible. Some of thecomments agree that in software development a product backlog is important, however itdoes not have to be very detailed since it can be improved during the project. Moreover,the respondents agree that it is important that the backlog is prioritized so that it is clearto everyone what should be done next.

45

71% of the respondents considered customer benefit to be more important than customersatisfaction, opposed to 29% favoring customer satisfaction. The respondents consideredthis question hard to answer since both parameters are considered important. Several ofthem argued that customer satisfaction will be a result of customer benefit and thereforethis should be focused on. One of the comments addressed customer satisfaction to resultin short term successful projects and satisfied customer. The same respondent consideredaiming for customer satisfaction to be a defensive strategy that could sometimes hinderchanges to the project.

Table 3 shows the percentage of how often the respondents stop to think about thebenefit of the project during the project. Some of the comments agree that the reflectionfrequency depends on the project management in terms of formal projects with controlledorganizations versus projects where product owners and customer is close to the teamregarding both competence and location. Another aspect is the time since sometimesthere is not enough time to stop and think and focus is on finishing functionality. Someof the respondents say they would like to spend more time reflecting on the benefit of theproject.

Alternative PercentageSeveral times a day 14%Several times a week 50%Several times a month 14%Once a week 14%Once a month 0%Not very often 0%Never 0%Other 7%

Table 3: Frequency of reflection of benefit

Moreover, the average of the respondents’ grading of their experience of projects wheretools such as Google Analytics have been used to collect the behavior of the customer to3 out of 5. All of the comments summarize that the respondents have worked on projectswhere Google Analytics is used. However, they agree that it has not been used to itsfull potential in the projects. One respondent commented that when Google Analytics isused, it is only used to measure usability, however the respondent believes that it wouldbe more e↵ective to measure, through quantitative data, if changes result in increased ordecreased benefit.

On average the respondents considered the importance of using such a tool to measurebehavior to be 4.6 out of 5. All of the comments states that the respondents considerthis to be very important. They agree that only measurement will give the truth, and itis a prerequisite to learn and become better. Some of the respondents commented thatnot all projects are suitable for measuring with specific tools, however they still thinkother forms of measurements are necessary. Furthermore, some comments address thefact that customers have not yet appreciated the value of for example Google Analytics.In addition it is important to present the data in an understandable way.

46

Other comments regarded the aspect of being physically located in the same place. Ifthe project is geographically spread the spontaneous conversation, especially the oneregarding the cross-functionality, where project managers, UX designers, and developersinteract with each other is limited. The respondent points out that it is much better tobe physically located in the same place.

7.1.2 Customer Perspective

A total of four customer representatives answered the questionnaire, whereof two productowners, one client, and one requirement specifier working with A/B testing and technicalanalysis. All of the respondents considered the alternative ”the project fulfills the re-quested functionality” to be the most important parameter of the project. The averageof the respondents’ ratings of the importance of all the project parameters is displayed inTable 4. One of the respondents also provided comments regarding the rating, explainingthat functionality is often very much reduced and thus it was important that the initialfunctionality is still there. The budget was not considered to be such an issue since itseldom occurs gigantic things going far o↵ of budget.

Parameter Rating (out of 5)The project is completed on time 4The project is completed within budget 3.3The project fulfills the function required 5The project is a good investment for the customer 4

Table 4: Importance of parameters

The importance of a clear backlog was considered to be 4 out of 5 on average. Further-more, the importance of the team working with specific techniques was graded as 2.3 outof 5 on average. The respondents commented that they trust the team to choose suitabletechniques for them. Furthermore, as long as goals, productivity, and customer benefitis high the techniques used do not matter.

All of the respondents consider releasing before the project is finished to be a good thing.They point out advantages such as finding bugs early, creating and accomplishing a goodproduct from the beginning as well as testing taking place in smaller units instead of ina huge final release.

The customer representatives considered their experience of projects working with toolssuch as Google Analytics to be an average of 4.3 out of 5. One of the respondentscomments that they do use Google Analytics from time to time but they should alsomeasure both before and after the project. The same respondent says that they oftenmiss to follow-up with measurements later on. The importance of using Google Analyticsor similar tools was considered as 4.5 out of 5.

47

7.2 Summary of Interview Answers

The following is a summary of the information gained during the evaluation sessionsheld with the project representatives described in Section 4.5.1. The overall opinions aregrouped according to topic.

7.2.1 General Discussions

It was discussed that there has to be a willingness from higher up in the organization toperform measurements and find the best version of the website. Everyone talks aboutmeasuring things, but no one does it according to one of the interviewees. One of theproject 4 interviewees considered it essential to spend some time during the initiation ofthe project to explain to the customer why measurements are necessary and simply statethat this is how you do it, it is a standard way of working even though it may not be astandard just yet.

The project 2 interviewees considered it important that the customer is part of themeasurement process from the beginning because then the customer will understand whyand how the measurements are performed. However, often the customer does not wantto be part of the process, they believe that they get a product from a factory deliveredin a box. The project 3 interviewees also pointed out the importance of transparencywithin the project to get a collective understanding of where the project is heading.

One of the project 3 interviewees had been working on a similar solution to the oneproposed by this thesis work, although that process was more globally oriented for allservices incorporated by their business. The interviewee liked the proposal and thoughtit was reasonable in many aspects. One improvement suggested by the same intervieweewas to remove keywords related to Scrum since all agile teams do not have sprints intheir work process. Instead the process could include regular release and measure pointswhere the same activities are preformed as defined by the proposal so far.

Both the project 3 interviewees and the project 4 interviewees thought that the processcould be used with other measurement tools and frameworks beyond Google Analytics.The point is to keep to the cores of the process such as setting up the measurementfrom the beginning when all other environments are set up, to continuously follow-up onmeasurements and present them to stakeholders.

One of the project 2 interviewees said that the idea of Selenium tests is good howeverit is not very easy to implement and requires extra e↵ort. Therefore the motivation toimplement them would decrease if the only purpose is to generate dummy data.

7.2.2 Previous Experience of Measurement

The project 1 interviewees’ experience was that measurement had been a part of a ma-jority of their projects. Furthermore, their experience was that the front end developerswere the ones being responsible of the measurements. They had sometimes done someimplementation of tracking, however that also depended on the team structure. If the

48

team had overlapping responsibilities they had sometimes done some front end develop-ment as well as implemented tracking scripts. In their current project, i.e. project 1, theydid not even have access to view the Google Analytics account and thus did not have areason to check and follow-up data. Instead it was left to the front end developers.

The project 3 interviewees had some experience of projects where analytics was a part ofthe project, however they had not been directly involved in the measurements and theyliked the idea of continuous follow-up on implemented functionality and KPIs.

The project 4 interviewees really liked the part of setting up Google Analytics in thebeginning of the project and said they would implement that part immediately in theirnext project. Their experience was that often there exists a user story regarding imple-mentation of Google Analytics in the backlog, however with that setup it is possible forthe stakeholders to down-prioritize that story resulting in it being implemented late inthe project.

7.2.3 Importance of KPIs

Furthermore, to be able to run the project from measurements the interviewees consideredit incredibly important that the KPIs are defined from the beginning, as described in thepre-study and startup phase of the proposal. As one of the project 2 interviewees stated:

”The KPIs also need to be made apparent to the team members and the cus-tomer, and also that it’s visually clear that this is what we aim for. The mostimportant thing is not that it looks like this or works like this, but that weachieve these things.”

Therefore it is important that the team finds ways to achieve this and also continuouslyfollow-up on it. One of the interviewees pointed out that this should be a simple thing, itshould just be done. The same person believes that it is the knowledge that is insu�cientand that there is a lack of interest in pursuing the measurements.

7.2.4 Dashboard Screen

Several of the interviewees liked the idea of having a dashboard screen with macro con-versions. One of the project 2 interviewees had the experience that dashboards generateengagement and visibility of what is going on. By watching the dashboard screen failuresand other issues are solved as soon as they show up on the screen. Another benefit isthat it provides a window for the business, stakeholders and other interested people canpass by and get a quick understanding of the progress. The project 3 interviewees alsoliked the dashboard screen since it creates openness and motivates the team.

7.2.5 Hypothesis Writing and MVP

The project 4 interviewees considered all parts of the process to be feasible at Valtechtoday, apart from the MVP and hypothesis part. For this to work, and to approach a

49

Lean Startup and lean way of work would require another type of business agreementbetween Valtech and the customer. They suggested that the Lean Startup artifacts shouldbe kept as a separate part of the process for people to apply to the process proposal asa whole. Otherwise they saw the risk of people dismissing the proposal because they arenot confident working with MVPs.

The interviewees of project 1 were the only ones who got to rewrite user stories ashypotheses and their opinions were that it was forced and basically a rephrasing of theitems. None of them fancied the classical user story phrasing of ”as a ... I can ... so that...” either and preferred a set of activities instead. In their current project their backlogconsists of reported issues in GitHub. The project 4 interviewees agreed that hypothesiswriting might be hard. The project 3 interviewees on the other hand liked the idea ofwriting hypotheses instead of the classical user story style because it reduces the ”justdo it” feeling that often surrounds a user story.

One of the project 2 interviewees had some experience of working with MVPs. TheirMVP process consisted of the customer and product owner specifying what they wantedto be done at story level, and then the team would get the story and divide it into smallerparts that would in themselves result in value for the consumer. The value was followedup by one of the interviewees talking to the web analysts at the company, however thisfollow-up was not performed on a regular basis and was not tied to respective story. Thiswas something that the interviewee considered to be a very good idea. Furthermore, theinterviewee considered MVPs to be hard to perform since it is hard to take a small piecesince a piece in the final product is often large.

7.2.6 Backlog Refinement Session

The backlog refinement part of the project was approved of and one of the intervieweesstated the following:

”... it is rare to share refinement things between teams and projects, you arecareless with the meeting because nobody wants it, everybody wants to code,but every time at the sprint planning you regret it...”

Several questions that an interviewee brought up as useful to think of during the backlogrefinement meeting are ”how should we measure this site or function?”, in combinationwith the usual questions ”how should we test it?”, ”how should we solve it?”, and ”isit reasonably sized?”. One of the interviewees suggested adding the measurement to anacceptance criteria and also in the definition of done.

7.2.7 Presentation of Measurements at Demo

The project 2 interviewees pointed out that it needs to be clarified that the data showedat the sprint demo is the measurements performed in the sprint before the current sprint,since the release of the current sprint will take place after the demo. Naturally thisis based on continuous deployment. The interviewees liked the idea of presenting thisdata at the demo since it will provide a nice update since the last time. The project

50

4 interviewees agreed that the combination of a dashboard screen and the presentationat sprint demo would be very good to increase awareness and interest in measurements,both from team members’ and stakeholders’ point of view.

7.3 Conclusion of Improvements

The following is a summary of the information gained during the evaluation sessions thatwill directly a↵ect the proposed process:

• The customer should own the Google Analytics account

• Every team member of the team should have access to view the Google Analyticsaccount and the data being gathered

• The customer must be aware of changes from the beginning and be part of theprogress throughout the project

• Words specifically related to a certain agile method could be removed or corres-ponding alternatives could be added

Table 5 shows where in the process improvements have been merged. Refer to AppendixC for the complete process.

Improvement Merged intoClarify that the customer should ownthe Google Analytics account

project startup - sprint 0

Ensure that every team member of theteam has access to view the GoogleAnalytics account

project startup - sprint 0

Ensure that the customer is aware ofchanges from the beginning and is partof the progress throughout the project

ongoing project - sprint 1-N

Remove words specifically related toa certain agile method or add corres-ponding alternatives

ongoing project - sprint 1-N

Table 5: Improvements of process

In excess of the improvements in Table 5, two other important statements that cameforward during the evaluation sessions where the importance of transparency of inform-ation and a thorough documentation by the end of the project. The documentationshould contain thorough descriptions of how the customer should continue to work withmeasurements and follow-up.

51

8 Discussion

The aim of this thesis work has been to suggest a checklist or work flow for Valtech toimprove their work with website analysis by integrating the necessary tasks into the agilework that is one of the pillars at Valtech. To do this, the Lean Startup in Section 3.1 andLean UX in Section 3.2 were studied. These are well-known concepts at Valtech, manyemployees work with UX and Je↵ Gothelf even gave a presentation at Valtech’s o�ce inApril 2014 [47].

8.1 Research Questions Revisited

8.1.1 Use of Measurements Between Sprints

To work with website analysis in an agile manner presupposes that data gathered duringthe sprints or a specified number of weeks, is followed up iteratively in the agile soft-ware development rhythm. The process I propose contains some milestones where themeasurements should play a central part, namely at the prioritization sessions, refine-ment sessions, planning sessions, and the demo session as well as continuously follow-upmeasurements after the project completion. The earlier the team starts to discuss howthey should measure the things they implement, the better.

My impression is that a very important use of the continuous measurements review is thatboth the team and the customer will realize the necessity of website analysis. Merely thefact that data will be looked upon continuously will be a significant improvement fromtoday.

8.1.2 Team Composition

One of the research questions concerned the team composition. My contemplating re-garding this question was whether and in that case how the team composition wouldchange with the integration of website analysis. During the pre-study interviews it wasdiscussed that the person or role necessary for the measurement and analysis should be aUX person. During the evaluation interviews also the front end developers where adver-ted as suitable and responsible. Some of the pre-study interviewees also pointed out thata mixture of UX and front end knowledge would be necessary. I also contemplated thepossibility of bringing a web analysts into the team at needed occasions only. However,during the interviews it was made evident that this was not the preferred solution.

From the pre-study and the evaluation it could be concluded that someone should beresponsible for ensuring that website analysis is performed and that the suitable stepsof the process defined in this thesis is followed. However, it should be noted that thisperson does not have to be the one to implement all the tasks and making all decisionsbased on the measurements. It is important that it is not a separate person who comesin that only works with the analysis. Although, several interviewees pointed out that itis necessary that expert knowledge is available and accessible to the team.

52

In theory, website analysis may appear as an easy task, simply gather data, then lookat it, and then make reasonable decisions. However, my impression is that to answerthe question ”what should we measure?” is not easy, neither the customer nor the webanalyst know the answer immediately. I believe that the actual website analysis requiresexperience even though the task of implementing necessary tracking scripts is not hard.Therefore I think that if it is possible to raise the general level of knowledge at Valtechand get more people interested and involved in the website analysis we will get far.

Another reason why I consider it necessary that the whole team is involved in the datagathering and analysis, is that the developers of the team must be aware of the trackingscripts so they do not end up ruining the tracking by mistake if they edit the code. If thegeneral participation increases, the engagement will increase resulting in website analysisbeing a natural part of the project work. Also, one of the pillars of agile and Scrumis to avoid having specific team members that the project depends on, thus the desiredcross-functionality.

8.1.3 The Issue of Early Release

The issue of early release has been present throughout the whole thesis work since itis impossible to gather user data if there are no users because the product is not yetreleased. When to release the product is very much a decision of the customer andaccording to the people talked to during the thesis work most customers do not wantto release their website until it is finished. Although only 4 customer representativescompleted the survey during the evaluation phase it is interesting that all of them werepositive to releasing the product early even though it was not finished.

The Lean Startup revolves around the build-measure-learn feedback loop and aims toshorten it as much as possible. Furthermore, to build MVPs are essential to gain validatedlearning and being able to create successful products without wasting time, money, andresources. Without the continuous deployment it is hard to work fully according to theLean Startup, however it is still possible to cherry-pick suitable parts.

During the interviews di↵erent types of projects were discussed as well as what could bedone to make customers more willing to release early. It was noted that Valtech’s projectsmore often consist of information websites than e-commerce websites. Furthermore, howfar it is possible to follow the Lean Startup also depends on whether the project consists ofnewly development, meaning that the website is being built from scratch, or reformationdevelopment, meaning that an existing website is being rebuilt and improved.

It could be argued that continuous release is better suited for reformation developmentsince it is easier to rebuild and release parts of the website while keeping the old versionsavailable. This is a scenario I believe is common in website development today as well.In that case it is possible to treat each small part of the website reformation as an MVPand iterate the releases to continuously gather data and learn from the process.

Moreover, when developing new websites from scratch it is common to develop for quite along time before it is possible to perform the first release. This is probably due to the factthat before a certain point there is not much to release. Naturally, this is not always the

53

case but it seems to be a common way of work. However, I still believe that it is possibleto speed up the first release partly by reforming the customer’s attitude. Another thingwould be to release beta versions of the product. The beta versions can then be treatedas MVPs.

The attitude of early release may also be inflicted by developers and team membersbuilding the website since they may have objections and opinions as well as the matter ofhabit to not advocate the MVP and early release strategy. It is probably also dependenton the team’s awareness of website analysis and how important they consider it to be.

If the customer project is an e-commerce site then performing measurements and identi-fying KPIs and what conversions to track is relatively easy since the aim of the websiteis to sell products or services. However, many of Valtech’s customers are customers thatdo not have their website as the main channel of their business. Their websites exist toshare knowledge and information, thus to identify conversions is harder.

8.2 The Proposal

8.2.1 General Di�culties

One of the di�culties I encountered throughout the thesis work was the di�culty ofkeeping objective when suggesting this process. It was necessary to keep a distance tomy own opinions of what is a good work process as well as an open mind to people thatI have interviewed and talked to during my work.

In continuation, something I noted during the work was the di↵erence between a processand a checklist. A process is a particular course of action intended to achieve something,whereas a checklist is nothing more than a list of items to be checked or consulted. Inother words, a process is more explicit in defining how something should be done and achecklist is simply telling you that something should be done, not how.

It is not always explicitly defined how something should be achieved in my proposal, sinceit is supposed to be of guidance and not a set of rules to follow. This is also something Iconsider an essential aspect of agile methodologies: they should facilitate your work andbe adaptive in terms of the team applying what they need from the framework.

8.2.2 Analysis Already Being Performed

The knowledge gathered through this work indicates that some measurement is part ofmost of the projects at Valtech. However, this does not imply that the development isdriven by learning and statistics. Furthermore, according to the web analysts at Valtechthis is not enough and their work is not easily performed as the setup is today. Selectivemeasurements are performed but are not followed up. Impact goals fall into oblivion andare seldom reviewed. However, as was discussed during the interviews the follow-up ofimpact goals is often not part of a consultant’s tasks. When the project is finished thecustomer is happy because they have a brand new and good looking website, but often

54

they forget or leave out the review of the impact of the project. In order for Valtech tobe market-leading in their field they need to improve this.

Although this may sound simple, it may not be because in the end it is the customerwho is the one to receive the product and thus the one responsible for maintaining andfollowing up on impact goals. Valtech may perform very well with continuous measure-ments throughout the project, but still, the customer needs to be a part of the progressand be ready to continue where Valtech diverges.

8.2.3 Possible Obstructions

During the thesis work I have come across some characteristics that may become an issuewhen working according to my proposal. It is sometimes the case that teams working onthe same project consist of both consultants from Valtech and third party consultants, aswell as internal developers of the customer. Because of this mixture it is necessary thatthe team still complies or agrees to comply with the process or a similar way of work asI propose.

In addition, team members may be geographically spread which will sometimes reduce orobstruct the agile artifacts of for example Scrum or Kanban, as well as other frameworksor processes the team would like to follow. Furthermore, this may a↵ect the process insuch a way that the team is harder to organize and therefore it may also be harder tokeep to all steps in the process. However, it will still be possible to perform several ofthe steps independently of location.

There may be internal policies or so called internal political issues at the customer sideresulting in specific decisions being made, specific reasons for not performing release, or towork with specific techniques and so on. As discussed in Section 8.1.3 di�culty achievingearly release in the project might be an obstruction. Such di�culties may result in theteam having to spend a lot of time trying to convince the customer and thus has tostruggle to build MVPs.

Lastly there is the aspect of everybody working in their own way, i.e. you may not beable to make people work in a certain way because they will still keep to their viewsand intentions. Naturally this is a good thing because it means that the team will coverdi↵erent issues and be able to solve problems through di↵erent experiences and views.However, my point is that in order for people to accept this proposal or way of work,the steps included needs to be simple and easy to embrace. These characteristics arecommon and I believe that a thorough evaluation as the one describe in Section 8.3.2would propose solutions to many of them.

8.2.4 Alternative Solutions

It may be the case that a checklist is not the best solution to achieve agile website analysis.However, I believe that it is an e↵ective way to get started. Maybe it would have beenconvenient to do something similar to U-Scrum [32], but I consider it to be a little tooruling for a company like Valtech. Also it is based on Scrum in such an extent that

55

it requires the project to use Scrum and that would mean locking oneself to a certainmethodology. A strength of my proposal is that it is possible to remove all keywordsrelated to certain methodologies and thus it is adaptable to the context.

It may also be the case that the Lean Startup [34] and Lean UX [36] are not necessarilythe way to go. They were proposed as good reading early in the project work by mysupervisor and therefore I started studying them. An important aspect is that both theLean Startup and Lean UX are rather well-known at Valtech and therefore it could beeasier for people to accept the process proposal if they can relate to it.

It could be argued that the process proposal should be more extensive in terms of ex-plaining in detail how the steps will be achieved. But then again, it would become lessadaptable and more ruling which is not what I aim for.

8.3 Method Criticism

8.3.1 The Interviews

Apart from the literature study, a central source of information has been the interviews.It is di�cult to think of suitable interview questions that are not leading the intervieweeon to much but still result in the essential information. The interviews were conductedwith a limited set of employees, thus only their opinions were gathered as stated in Section5. Furthermore, the sessions may have a↵ected each other in such ways that questionswere altered between interviews, some interviews led to new questions being posed in thesubsequent interviews and so on.

In retrospect, the fact that there are not that many people with an explicit web analyticsand strategy function at Valtech was a di�culty since it would have been convenient totalk to several web analysts and strategists during both the pre-study and the evaluationpart of the thesis work. It might also be the case that there are actually several peoplewith the necessary knowledge at the company, however I did not come across them duringthe work.

8.3.2 The Evaluation

The evaluation part where team members were interviewed and discussed the proposalwas worthwhile although not comprehensive. To completely evaluate the process it isnecessary to apply it on several projects during a long term. I believe that to get enoughtime to apply the process and learn from doing it while testing new variations, it wouldbe necessary to perform the process approximately during a year. The process will thenbe iterated through the development process and the team members as well as customerrepresentatives will notice what parts work well and what parts need to be improved. Itmay also be the case that the di↵erent steps apply to di↵erent projects since everythingis very dependent on the context, such as budget, customer attitude, team structure andlocation to name a few. The contextualization was also something that came forwardduring the pre-study interviews.

56

8.3.3 A Status Inventory

To get a status on how far the teams at Valtech have come in their work with websiteanalysis, an inventory of the status of website analysis and the use of Google Analyticsand similar tools in the beginning of the thesis work would have been helpful. As asuggestion this could have been done by performing a survey among all the employeeswith questions addressing their experience of working with collection, measurement, andanalysis of user data. Without this inventory this knowledge was harder to achieve, whichwas done by simply talking to people. While this surely gave an indication of the status,a survey would have been more comprehensive.

8.3.4 Compliance With the Agile Manifesto

In Section 2.2 I quoted the Agile Manifesto [16] which favors individuals and interactionsover processes and tools. At first it may seem contradictory to my work of expanding theagile framework with a process of work. I believe that my proposal is not an obstructionagainst the manifesto since it is basically a getting started guide for Valtech to becomesuccessful in their website analysis. In continuation I believe that the major point of theagile manifesto is to get away from all the bureaucracy that project management maybring about that may hinder the development process more than improving the projectprogress.

8.4 Website Analysis and Ethics

Among all interviews there was only one interviewee who mentioned the aspect of ethics.However, it should be mentioned that no questions directly addressed that topic. I believethat the ethical part of the data gathering is not something that the consultants spendvery much thought on. Their purpose of measurements is to improve the user experienceand the benefit of the consumer and thus their intentions are good. That is why the UXpeople spend so many hours on user tests as well.

I believe that similar to the discussion presented in Section 2.1.2 people do not considertheir data gathering as possibly harmful to the consumer. I do not think so either, howeverthere is the possibility to use the data in an unethical way and therefore it is importantthat team members, customers, and stakeholders are aware that with knowledge comesresponsibility.

57

9 Conclusions

9.1 People and Characteristics

One of the foundations of this proposal is that someone wants to perform the measure-ments in the project and be passionate about it. There needs to exist an interest thatdrives the measurements even though there exists a checklist for what to do. The atmo-sphere at Valtech is free meaning that as an employee you can work freely and creativelyand thus employees want as few rules as possible. In addition, teams change all the timewhich may generate di�culties since people are replaced with new colleagues and ways ofwork may change. Moreover, the customer needs to be an active customer, they need tolearn so that they can move forward after the project and be independent in their ana-lysis of user behavior. It is also important to follow up quantitative measurements withqualitative measurements where specific questions are asked to learn why the statisticslooks the way it does. In other words, website analysis and quantitative data can neverreplace qualitative data.

9.2 Lean Startup, Lean UX, and Agile Website Analysis

During my work I have come to the conclusion that Lean Startup and Lean UX is nota requirement for working with website analysis early in the project and in an agilemanner. However, website analysis is a tool and a necessity to work according to theLean Startup and Lean UX as well as in achieving a working statistical and measurementdriven development process. Thus, agile website analysis is a step forward towards suchan organization.

9.3 Research Questions

To once again review the research questions, website measurements will be used betweensprints by continuously follow-up on the data, learning from it, and acting upon it.Furthermore, increased awareness of the importance and relevance of spending more timeand focus on website analysis and measurements will be achieved through the regularfollow-up on measurements that will be part of the demo for stakeholders.

The agile team composition will not change very much from today. The focus of cross-functional teams that already exist at Valtech will continue. However, one more respons-ibility will be added to the cross-functionality, namely the responsibility of making surethat website analysis is part of the project work from the beginning by following suitablesteps from the process proposal of this thesis work.

Early release could be achieved by performing beta releases and trying to split up workin MVPs. Another aim should be to build trust in Valtech’s ability to deliver projects inless time and money by continuous deployment and validated learning through websiteanalysis.

58

9.4 A Minimum Viable Process

A minimal process that Valtech needs to adapt as early as possible in their projectscontains a subset of the process presented by this thesis work. Google Analytics orsimilar statistics tools need to be set up in parallel with the setup of other necessaryenvironments such as development and test environments. The dashboards and customreports identified by the project should be set up in the statistical software. During thisproject startup and beginning of software development the measurements should be testedto some extent in order to see if the intended data is gathered. Lastly, the dashboardscreen should be put up in addition to the regular build screen that is standard to most ofthe development teams at Valtech. Together with this everyone should aim for continuousdeployment as soon as possible.

9.5 Recommendations for Valtech

The topic of recruiting more people with knowledge and experience of website analysisarose several times, both during interviews with the existing web analysts and duringthe evaluation sessions. It could be useful for Valtech to try and recruit such people tofurther improve their analytics ability, however it is reasonable to try out the processproposal first.

9.6 Future Work

A master thesis is too limited in time to perform a comprehensive evaluation of theproposal. Thus the future work necessary to perform includes further evaluation of theprocess as described in Section 8.3.2 where the process is applied to several projectsresulting in suitable adjustments being performed. Also, the questionnaire to determineproject measurability used in the pre-study phase of the process needs to be refined. Thisrefinement will probably be a natural e↵ect of applying the process and thus questionnaireon several projects.

Future work also includes to fully approach a Lean Startup organization and agree uponsuch business agreements with the customers. If budgets and tightly controlled organiza-tions turn our to be a problem for the process, then having a look at beyond budgeting [48]founded by Bjarte Bogsnes may be helpful.

During the evaluation of the process several project members discussed the possibilities ofthe process and wanted to remove the limitation of Scrum, Kanban, and Google Analytics.Thus in the future it would be interesting to apply the process to other similar issues.

59

References

[1] Google Analytics. http://www.google.com/analytics/. Retrieved July 15, 2014.

[2] Daniel Waisberg and Avinash Kaushik. Web Analytics 2.0: Empowering CustomerCentricity. The original Search Engine Marketing Journal (SEMJ.org), 2, 2009.

[3] WP Dev Shed. Importance of Web Analytics. http://wpdevshed.com/importance-of-web-analytics/, 2013. Retrieved April 16, 2014.

[4] Web Analytics Association. Web Analytics Definitions. http://www.digitalanalyticsassociation.org/Files/PDF standards/WebAnalyticsDefinitions.pdf,2008. Retrieved April 28, 2014.

[5] Avinash Kaushik. Best Metrics For Digital Marketing: Rock Your Own And RentStrategies. http://www.kaushik.net/avinash/best-web-metrics-digital-marketing-own-rent-strategies/, 2014. Retrieved June 16, 2014.

[6] P.M. Schwartz. Privacy, Ethics, and Analytics. Security Privacy, IEEE, 9(3):66–69,2011.

[7] Google. Google Analytics Terms of Service. https://www.google.com/analytics/terms/us.html. Retrieved June 13, 2014.

[8] Frank Buytendijk. The Ethics of Analytics: A New Perspective? http://radiantadvisors.com/2012/12/06/the-ethics-of-analytics-a-new-perspective/, 2012.Retrieved June 10, 2014.

[9] About Frank Buytendijk. http://www.frankbuytendijk.com/bio.htm. Retrieved July16, 2014.

[10] About Unic. https://www.unic.com/en.html. Retrieved July 16, 2014.

[11] Lukas Oldenburg. Google Analytics! So you’re a spy, right? http://www.webanalyticsworld.net/2012/10/google-analytics-so-youre-a-spy-right.html,2012. Retrieved June 13, 2014.

[12] About Lukas Oldenburg. http://www.webanalyticsworld.net/about-us/featured-bloggers#LukasOldenburg. Retrieved July 16, 2014.

[13] Digital Analytics Association. The Web Analyst’s Code of Ethics. http://www.digitalanalyticsassociation.org/codeofethics. Retrieved June 10, 2014.

[14] Lamont Wood. Analytics Pros See Value in Code of Ethics for Grow-ing Field. http://data-informed.com/analytics-pros-see-value-in-code-of-ethics-for-growing-field/, 2013. Retrieved June 10, 2014.

[15] Osama Sohaib and Khalid Khan. Integrating usability engineering and agile softwaredevelopment: A literature review. In Computer Design and Applications (ICCDA),2010 International Conference on, volume 2, pages 32–38, 2010.

[16] Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham,Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Je↵ries, Jon

60

Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Je↵ Sutherland,and Dave Thomas. Manifesto for Agile Software Development. http://agilemanifesto.org/. Retrieved June 27, 2014.

[17] Torgeir Dingsøyr, Tore Dyba, and Nils Brede Moe. Agile Software Development:Current Research and Future Directions. Springer, 2010.

[18] Ken Scwaber and Je↵ Sutherland. The Scrum Guide. https://www.scrum.org/Portals/0/Documents/Scrum%20Guides/2013/Scrum-Guide.pdf, 2013. RetrievedApril 14, 2014.

[19] Muhammad Ovais Ahmad, Jouni Markkula, and Markku Ovio. Kanban in SoftwareDevelopment: A Systematic Literature Review. In Software Engineering and Ad-vanced Applications (SEAA), 2013 39th EUROMICRO Conference on, pages 9–16,2013.

[20] Tore Dyba and Torgeir Dingsøyr. Empirical studies of agile software development: Asystematic review. Information and Software Technology, 50(9–10):833 – 859, 2008.

[21] Mary Poppendieck. Lean Software Development. In Software Engineering - Compan-ion, 2007. ICSE 2007 Companion. 29th International Conference on, pages 165–166,2007.

[22] David J Andersson. Kanban: Successful Evolutionary Change for Your TechnologyBusiness. Library of Congress Cataloging-in-Publication Data, 2010.

[23] Trello - Organize anything, together. https://trello.com. Retrieved July 1, 2014.

[24] JIRA - Issue & Project Tracking Software. https://www.atlassian.com/software/jira.Retrieved July 1, 2014.

[25] Gary Angel. Thinking Agile: Thoughts from X Change. http://semphonic.blogs.com/semangel/2012/09/thinking-agile-thoughts-from-x-change.html, 2012. Re-trieved June 11, 2014.

[26] About Gary Angel. http://semphonic.blogs.com/about.html. Retrieved July 16,2014.

[27] About Himanshu Sharma. http://www.optimizesmart.com/about-us/. RetrievedJuly 16, 2014.

[28] Himanshu Sharma. How to use Agile Analytics to quickly solve your Conver-sion problems. http://www.optimizesmart.com/use-agile-analytics-quickly-solve-conversion-problems/. Retrieved June 11, 2014.

[29] Minna Isomursu, Andrey Sirotkin, Petri Voltti, and Markku Halonen. User Experi-ence Design Goes Agile in Lean Transformation - A Case Study. In Agile Conference(AGILE), 2012, pages 1–10, 2012.

[30] Jennifer McGinn and Ana Ramırez Chang. RITE+Krug: A Combination of UsabilityTest Methods for Agile Design. Journal of Usability Studies, 8:61–68, 2013.

61

[31] The Agile Research Network, LShift Ltd., Helen Sharp, Laura Plonka, PeggyGregory, and Katie Taylor. Integrating UX design into a DSDM project: chal-lenges, working practices and lessons learned. http://agileresearchnetwork.org/wp-content/uploads/2013/11/DSDM UX White Paper-update-151113.pdf, 2013. Re-trieved April 24, 2014.

[32] Mona Singh. U-SCRUM: An Agile Methodology for Promoting Usability. In Agile,2008. AGILE ’08. Conference, pages 555–560, 2008.

[33] Desiree Sy. Adapting Usability Investigations for Agile User-centered Design. Journalof Usability Studies, 2:112–132, 2007.

[34] Eric Ries. The Lean Startup: How Today’s Entrepreneurs Use Continuous Innova-tion to Create Radically Successful Businesses. Crown Business, 2011.

[35] Beverly May. Applying Lean Startup: An Experience Report – Lean & Lean UXby a UX Veteran: Lessons Learned in Creating & Launching a Complex ConsumerApp. In Agile Conference (AGILE), 2012, pages 141–147, 2012.

[36] Je↵ Gothelf with Josh Seiden. Lean UX: Applying Lean Principles to Improve UserExperience. O’Reilly Media, 2013.

[37] Tomas Jansson and Lennart Ljung. Projektledningsmetodik. Studentlitteratur, 2004.

[38] Johan Hyden and Rutger v Scheele. E↵ektmalet - det viktigaste malet av alla.Projektvarlden, 3, 2013.

[39] Ingrid Ottersten and Mijo Balic. E↵ektstyrning av IT - Nyttan uppstar i anvand-ningen. Liber AB, 2004.

[40] Gojko Adzic. Impact Mapping: Making a big impact with software products andprojects. Provoking Thoughts Limited, 2012.

[41] Google. About Real-Time. https://support.google.com/analytics/answer/1638635?hl=en. Retrieved April 17, 2014.

[42] Google. How Analytics impacts your website code. https://support.google.com/analytics/answer/1008009?hl=en&ref topic=1008008. Retrieved April 17, 2014.

[43] Google Developers. Tracking Site Activity. https://developers.google.com/analytics/devguides/collection/gajs/asyncTracking. Retrieved April 17, 2014.

[44] Google Developers. Universal Analytics Upgrade Center. https://developers.google.com/analytics/devguides/collection/upgrade/. Retrieved May 26, 2014.

[45] Google. About Dashboards. https://support.google.com/analytics/answer/1068216?hl=en. Retrieved June 11, 2014.

[46] Google. About Custom Reports. https://support.google.com/analytics/answer/1033013?hl=en. Retrieved June 11, 2014.

[47] Je↵ Gothelf. Inspirationslunch: Building successful innovation teams. https://valtech.se/events/building-successful-innovation-teams/. Retrieved July 3, 2014.

62

[48] Beyond Budgeting Institute. http://www.bbrt.org/. Retrieved June 17, 2014.

63

A Pre-study Interview Questions

A.1 General Information Interviews

A.1.1 Agile and General Work at Valtech

1. Beratta kort om dig sjalv

2. Hur ser processen ut infor ett nytt projekt?

3. Vad ar det forsta som hander?

4. I vilken utstrackning ar Valtech projektledare for projekten som vi har?

5. Hur ser forstudien av ett projekt ut?

6. Vilka aktiviteter och moment forekommer da?

7. Hur arbetar man med e↵ektmal infor ett projekt?

8. Hur ser uppstarten av ett projekt ut?

9. Hur arbetar man med e↵ektmal under uppstarten av projekt?

10. Hur arbetar man med e↵ektmal under sjalva projektets gang?

11. I vilken utstrackning foljer man upp e↵ektmal efter projektet ar live?

12. I vilken utstrackning arbetar Valtech agilt?

13. Vilka agila arbetssatt praktiseras pa Valtech?

14. I projekt dar anvandartester (tex UX) ingar, hur anvands informationen som kom-mer fran dem i den agila processen?

(a) Scrum: sprint planning

(b) Scrum: backlog refinement

15. I vilken utstrackning efterfragar kunder matning och analys av kvantitativ data isina produkter?

A.1.2 User tests at Valtech

1. Beratta kort om dig sjalv

2. Kan man saga att anvandartesterna inom UX samlar kvalitativ anvandardata?

3. Vad skiljer anvandartesterna inom UX och den information om anvandarbeteendesom de samlar fran den information om anvandarbeteende som samlas fran exem-pelvis Google Analytics och klick-matningsverktyg pa webbsidor?

4. Hur arbetar Valtech med anvandartester idag?

5. I vilken utstrackning sker anvandartester?

64

6. Hur sker anvandartester i borjan av projektet?

7. Hur dras lardomar fran anvandartesterna?

8. Hur integreras dessa lardomar i arbetet?

9. I vilken utstrackning sker anvandartester pa en agil niva?

10. I projekt dar anvandartester ingar, hur anvands informationen som kommer frandem i den agila processen?

(a) Scrum: sprint planning

(b) Scrum: backlog refinement

11. Hur integrerar ni anvandartester med resten av utvecklingsteamet?

12. Jag har last the Lean Startup och inspirerats av MVP, experiments, build-measure-learn, MVT, i vilken utstrackning arbetar ni med sadant i era projekt?

13. I vilken utstrackning korrelerar anvandartesterna med e↵ektmal?

14. Hur skulle man kunna utforma anvandartester for att mata e↵ektmal?

15. I vilken utstrackning anvands e↵ektmalskartan?

16. Hur skulle man kunna utforma anvandartester for att mata delmal i e↵ektmalskartan?

A.2 Project Manager Interviews

1. Beratta kort om dig sjalv

2. I vilken utstrackning arbetar du med e↵ektmal?

3. Hur viktigt ar det att e↵ektmalet ar ordentligt formulerat infor projektet?

4. Vad hander med avseende pa e↵ektmalet nar projektet ar avslutat och produktenlevererad?

(a) Hur mater man det?

(b) Vem mater det, kundens ansvar?

5. I vilken utstrackning anvander du e↵ektmalskartan?

6. Hur fungerar e↵ektmalskartan i det agila arbetssattet?

7. Vilka erfarenheter har du av kvantitativ matning av anvandardata i projekt?

8. I vilken utstrackning efterfragar kunder matning och analys av kvantitativ data isina produkter?

9. Hur ser du pa att integrera kvantitativa matningar pa webbsidor i den agila pro-cessen?

(a) Scrum: sprint planning

65

(b) Scrum: backlog refinement

10. Utover att samla in data tidigt (fore release), vilka svarigheter ser du med en sadanintegration.

11. Jag har last en del om lean startup, MVP, MVT, experiment och build-measure-learn filosofin. Ar det nagot du har arbetat med eller kanner till?

12. Hur brukar releaseforfarandet se ut? Dvs nar brukar den tidigaste releasen tillproduktion goras? Endast nar projektet ar slut?

13. Vad hander efter forsta releasen till produktion?

14. Brukar det vara aktuellt att slappa beta-versioner? Varfor, varfor inte?

A.3 Web Analyst Interviews

1. Beratta kort om dig sjalv

2. Vad innebar statistikdriven design for dig? Vilka fordelar ser du med det?

3. Vad innebar larandedriven design? Vilka fordelar ser du med det?

4. Hur skiljer sig statistikdriven och larandedriven design fran data driven design?

5. Hur arbetar man med statistik och webbanalys pa Valtech idag?

6. Vilka behov finns for att det ska ga att genomfora?

7. I vilken utstrackning efterfragar kunder matning av sina produkter?

(a) Ar det nagot som de vill ha eller ar det nagot som vi foresprakar som ett sattatt arbeta och som da bor inga som en “service”?

(b) Borde det bli en del av var tvarfunktionalitet i teamen?

8. De moment i scrum som jag har identifierat som viktigast att integrera matningoch analys i ar backlog groooming och sprint planning, ar det nagon annan del avden agila processen som du anser vara viktig?

9. I vilken utstrackning jobbar du, och aven Valtech i stort, med e↵ektmal och ef-fektmalskartan?

10. Du foresprakar statistikdriven design och the Lean Startup-filosofin, kan du berattamer om hur du jobbar/forsoker jobba med tex MVP, experiment, validering, build-measure-learn osv? (Specifik fraga till en av intervjupersonerna)

11. Vilka svarigheter finns det med att jobba pa det har sattet?

12. Vilka fordelar finns det med tidig release, utover att man far nagot att mata pa?

13. Hur tidig release bor man gora for att det ska vara vardefullt?

14. Brukar det vara aktuellt att slappa beta-versioner?

66

15. Vad ar mest gangbart, att alla lar sig att jobba med matning och analys eller attnagra personer med extra kunskap kommer in och hjalper till pa sprintmoten ochdylikt?

A.4 Software Developer Interviews

1. Beratta kort om dig sjalv

2. Kan du beratta lite om hur ni har arbetat med sparning i ert nuvarande projekt?

3. Hur lange brukar ni fa vanta pa resultat av matningar?

4. Hur ar det att arbeta med matning och tanka pa analys?

5. Hur mycket analys gor ni?

6. Hur tycker du att ert arbete med sparning fungerar? Ar det tex tillrackligt ochskulle ni kunna gora mer?

7. I vilken utstrackning upplever du att Valtech arbetar med Google Analytics ellermotsvarande for matning?

8. Sag att vi har statistik om de saker som teamet ska utveckla, nar och hur tror duatt man bor/kan ta in det i arbetet? (Statistiken kommer inte fran kunden, utanfran matningar och analys som har gjorts av teamet)

9. Hur tror du att analysen av user stories ska goras? Forutsatt till exempel att dettar 30 dagar att samla in data och sedan 4 h att gora analysen.

10. Vilka personer tror du behovs for att kunna genomfora ett sadan har arbetssatt?Dvs, vilken teamuppsattning tror du skulle behovas?

11. Vilka delar av det agila arbetssattet, tex sprint planering i Scrum, tror du ar viktigaatt utoka med aktiviteter som en del av processen for integration av matning ochanalys?

12. Ser du nagra svarigheter med att arbeta med matning och analys kontinuerligtunder projektet?

67

B Questionnaires

B.1 Team Perspective

1. Vad ar din huvudsakliga profil? (UX, front-end, back-end, projektledare, annat)

2. Vilket projekt jobbar du pa just nu?

3. Vilken av foljande parametrar ar viktigast for dig?

(a) Att projektet ar klart pa utsatt tid

(b) Att projektet ar fardigstallt inom budget

(c) Att projektet uppfyller den efterfragade funktionen

(d) Att projektet ar en god investering for kunden

(e) Annan

4. Hur viktigt tycker du att det ar projektet ar klart pa utsatt tid? (Skala 1-5, juhogre si↵ra desto viktigare)

5. Hur viktigt tycker du att det ar projektet ar fardigstallt inom budget? (Skala 1-5)

6. Hur viktigt tycker du att det ar projektet uppfyller den efterfragade funktionen?(Skala 1-5)

7. Hur viktigt tycker du att det ar projektet ar en god investering for kunden? (Skala1-5)

8. Hur viktigt tycker du att det ar att det finns en tydlig backlog? (Skala 1-5)

9. Vad tycker du ar viktigast, kundnojdhet eller kundnytta?

10. I hur stor utstrackning stannar du upp och reflekterar over nyttan med ett projektunder projektets gang?

11. Hur stor erfarenhet har du av projekt dar man anvander verktyg som Google Ana-lytics for att mata besokares beteende? (Skala 1-5)

12. Hur viktigt tycker du att det ar att mata besokares beteende med exempelvis GoogleAnalytics? (Skala 1-5)

B.2 Customer Perspective

1. Vilket projekt jobbar du och Valtech pa just nu?

2. Vad har du for roll/arbetsuppgift i projektet?

3. Vilken av foljande parametrar ar viktigast for dig?

(a) Att projektet ar klart pa utsatt tid

68

(b) Att projektet ar fardigstallt inom budget

(c) Att projektet uppfyller den efterfragade funktionen

(d) Att projektet ar en god investering for kunden

(e) Annan

4. Hur viktigt tycker du att det ar projektet ar klart pa utsatt tid? (Skala 1-5, juhogre si↵ra desto viktigare)

5. Hur viktigt tycker du att det ar projektet ar fardigstallt inom budget? (Skala 1-5)

6. Hur viktigt tycker du att det ar projektet uppfyller den efterfragade funktionen?(Skala 1-5)

7. Hur viktigt tycker du att det ar att projektet ar en god investering? (Skala 1-5)

8. Hur viktigt tycker du att det ar att det finns en tydlig backlog? (Skala 1-5)

9. Hur viktigt tycker du att det ar att utvecklingsteamet arbetar med vissa tekniker?(Skala 1-5)

10. Vad anser du om att release till produktion sker ofta, aven om slutprodukten intear klar? (Skala 1-5)

11. Hur stor erfarenhet har du av projekt dar man anvander verktyg som Google Ana-lytics for att mata besokares beteende? (Skala 1-5)

12. Hur viktigt tycker du att det ar att mata besokares beteende med exempelvis GoogleAnalytics? (Skala 1-5)

B.3 Evaluation Interview Questions

1. Vilka ar ni? Beratta kort om er sjalva

2. Vad jobbar ni med for projekt just nu?

3. Har ni nagon matning idag?

4. Spontana tankar kring processen?

5. Hur mycket av det har kanns bekant eller som gors redan?

6. Ar det nagot som verkar svart, vad i sa fall?

7. Ar det begripligt?

8. Vad tror ni om team-konstellationen? Hur forandras den fran idag, vad behoverlaggas till i teamet?

9. Vem ansvarar for matningen idag?

10. Ar det nagonting som behover andras eller laggas till i processen?

69

C The Complete Process

C.1 Project Pre-study

1. Define business impact goals

2. Perform workshops with representatives from the client and Valtech

(a) Discuss and define KPIs

(b) Determine what conversions to track

3. Perform the questionnaire for evaluating the project measurability described insection 4.5.2

C.2 Project Startup - Sprint 0

4. Necessary KPIs should be defined if they are not already set

5. Create a Google Analytics account, owned by the customer, for the project

6. Set up the Google Analytics account in terms of

(a) Dashboards

(b) Custom reports

7. Show the client how to use the Google Analytics account to follow up data

8. Ensure that all team members have access to the Google Analytics account in termsof reading permission

9. In connection with set up of test and product environments, make sure that theminimal tracking code of Google Analytics is added as soon as the initial websiteis put up so that data will be gathered from the beginning and tracking will beperformed in every environment

10. Ensure that test environments are not indexed by Google searches

11. Put up a screen showing a Google Analytics dashboard of the three most importantmacro conversions based on the business impact goals identified in the pre-studyphase

12. Make sure that relevant product backlog items delivered during this sprint 0 arewritten as hypotheses (more about hypotheses in section C.3)

C.3 During Ongoing Project - Sprint 1 to Sprint N

13. Continuously ensure that the customer is up to date with measurements

70

14. If necessary, continuously educate customer representatives that will accede to themeasurements after project completion

15. Ensure that the dashboard configured in sprint 0 functions as intended, in otherwords make sure that the intended measurements are implemented. This could bedone by using special software that animates click through the website, such asSelenium tests and PhantomJS

16. Product backlog prioritizing

(a) Use the previous measurements from this project to prioritize the backlog

(b) Use knowledge and experience in terms of measurements from previous pro-jects

17. Backlog grooming (Scrum) or similar session - Think about analysis and how tomeasure user behavior for each backlog item discussed during the grooming

18. Sprint planning (Scrum) or similar planning session - Relevant user stories or back-log items should be written as hypotheses:

”We believe that [this capacity] will result in [this outcome].We will know we have succeeded when [we see a measurable signal]”

19. Build MVPs from the hypotheses above

20. Implement necessary measurements during implementation of the new functionality

21. Aim for continuous release as early as possible. In other words make sure to releaseto production after each sprint. Necessary sub steps for this is to set up a routinefor

(a) Continuous integration,

(b) Automated tests, and

(c) Continuous deployment

22. Sprint demo or similar demo session - Include measurements from the previoussprint to increase awareness of analytics. If sprints are not part of the projectorganization, set up a release schedule recurring with a few weeks in between.Define a similar scedule for when measurements should take place.

C.4 Project Completion

23. Make sure to create a strategy for further analysis

24. If further development is performed:

(a) Continue to release after every sprint or according to the release scheduledefined in step 22

(b) Add measurements to relevant new functionality, in consultation with the cli-ent or according to the planned strategy

71

25. Follow up measurements on a regular basis, as a suggestion once every month

72