software testing (in scala): a practitioner's survey (quickly)

Software Testing

Roberto Casadei{roby.casadei}@unibo.it

C.D.L. Magistrale in Ingegneria e Scienze InformaticheAlma Mater Studiorum—Universita di Bologna, Cesena

a.a. 2016/2017

R.Casadei (Universita di Bologna) Testing a.a. 2016/2017 1 / 47

Outline

Outline

Recap on unit tests in JUnit and TDD

Operative introduction to “automated builds with tests anddependencies” (in sbt)

Main kinds of tests: unit, integration, functional, stress, acceptance

Testing framework for Scala: ScalaTest

Test coverage

Introduction to test doubles and mocking

Some pointers to more advanced stuff on testingI Effective unit testingI Property-based testing (cf., QuickCheck, ScalaCheck)I Cucumber for behavior-driven developmentI Selenium for testing web apps via browser automationI Testing of GUI-based programsI Performance regression testing with ScalaMeter


Outline

1 Quick recap

2 sbt

3 Panorama on testing

4 Basics of testing in Scala with ScalaTest

5 Test doubles

6 More on testing


Recap: JUnit

1 public class DeviceImplTest {

2 private DeviceImpl device;

3

4 @Before public void init(){

5 this.device = new DeviceImpl ();

6 }

7

8 @Test public void on() throws Exception {

9 this.device.on();

10 assertTrue(this.device.isOn());

11 }

12 // ..........

13 }

Concepts

Test suite, test cases, tests

Assertions

Fixture

Arrange, Act, Assert


Recap: Test-Driven Development (TDD)

What

“Development” is guided (“driven”) by “tests”I I.e., it is not a testing activity; tests as (useful) side-effect

Exercise your code before even writing itI Makes you explore the problem before attempting a solutionI Forces you to think as the user of your codeI Makes you focus on what is important right now (cf., KISS)

Red – Green – Refactor cycle

Benefits

Specification before implementation

Guides the (detailed) design process

You end up with regression tests (i.e., tests to catch regressions)I A regression is when something correct gets brokenI A safety net against regressions is fundamental for refactoring activities


Outline

1 Quick recap

2 sbt



5 Test doubles

6 More on testing


The Scala Build Tool (sbt): intro

Goal: improve productivity through automation of project-related activities

sbt: the “standard” build tool for Scala

Belongs to the family of build tools, like Maven, Gradle and othersI Gradle is the one that will be covered in this course

Not limited to Scala projectsI Can seamlessly be used for Java projectsI Plugins for other languages (e.g., Groovy, Clojure, C++a, Jythonb..)

ahttps://github.com/d40cht/sbt-cppbhttps://github.com/markchadwick/jython-sbt-plugin

Lightbend Activator

A custom version of sbt which adds two extra commands, activatorui and activator new

Supports bootstrapping applications from a catalog of templates


https://github.com/d40cht/sbt-cpp

https://github.com/markchadwick/jython-sbt-plugin

The Scala Build Tool (sbt): basics

Key concepts

A build definition is defined in /build.sbt

It specifies a set of projects

Each projects is associated with a set of settings and tasks

Two modes of operation

1. Batch mode: $ sbt clean compile

2. Interactive mode

1 $ sbt

2 sbt > test

3 ...

4 sbt > console


The Scala Build Tool (sbt): basic usage

Directory structure (Maven-style)

1 projectRoot/

2 build.sbt

3 lib/

4 src/

5 main/resources/

6 main/scala/

7 main/java/

8 test/resources/

9 test/scala/

10 test/java/

11 project/

12 build.properties

13 plugins.sbt

14 target/

build.sbt

1 lazy val root = (project in file(".")).settings(

2 name := "mytest", version := "1.0", organization := "unibo.pps",

3 scalaVersion := "2.12.2",

4 libraryDependencies += "com.novocode" % "junit -interface" % "0.10

" % "test")


The Scala Build Tool (sbt): basic usage

Basic commands

help: shows help

settings: lists the settings for the current project

tasks: lists the tasks for the current project

<settingName>: shows the value of the setting

<taskName> ..args..: execute the task

Common tasks

clean: deletes files produced by build

compile: compiles sources

console: starts Scala interpreter with project classes in classpath

package: produces a main artifact (e.g., a JAR)

runMain <mainClass> ..args..

test: executes all tests


Outline

1 Quick recap

2 sbt



5 Test doubles

6 More on testing


Perspectives on testing1

“Schools” of testing

Analytic School: testing as a rigorous task, via formal methodsI Motto: Programs are logical artifacts, subject to math laws

Factory School: testing to measure project progressI Motto: Testing must be planned/scheduled/managed

Quality School: testing as a quality processI Motto: A quality process for a quality product

Context-Driven School: testing as a stakeholder-oriented, adaptableprocess

I Key question: which testing would be more valuable right now?

Agile School: testing to facilitate changeI Techniques: test automation, test-driven approaches

Key takeaway: perspectives on testing impact the what and how of testing

1https://www.prismnet.com/~wazmo/papers/four_schools.pdfR.Casadei (Universita di Bologna) Testing a.a. 2016/2017 12 / 47

https://www.prismnet.com/~wazmo/papers/four_schools.pdf

Many different kinds of testing

Multiple dimensions

Technology-facing vs. Business-facing testingI Verification vs. validation

Granularity/scope of testingI Unit – integration – system-level/end-to-end/acceptance

Static vs. dynamicI Code-level vs. runtime

Manual (human-based) vs. automated (computer-based)I Examples of human testing: inspection (code reviews), walkthroughI Examples of computer-based testing: static analysis, “standard” testing

Structural (aka white-box) vs. functional (aka black-box)I Examples of structural testing: control flow testing, path testingI Examples of functional testing: equivalence partitioning, BVA


Levels of testing

Levels of testing based on granularity/scope

1. Unit testing: testing at the level of individual units (of functionality)

2. Integration testing: testing of the functionality provided by multipleintegrated/interacting units

3. System testing: testing the whole system for correctnessI In a black-box wayI Non-functional requirements (e.g., efficiency, reliability, security) are

typically tested at this level

4. Acceptance testing: testing the whole system against the needs andexpectations of users and stakeholders


Acceptance tests

Acceptance testing

Is the SW system “acceptable” (for delivery)?

It’s about testing whether the SW system satisfies the acceptancecriteria (e.g., as specified in the requirements)

Acceptability is matter of customers, product owners, stakeholders, orusers

It is very much about what has to be built

⇒ Acceptance test-driven planning/development (ATDP/D)

Tools: FitNesse (wiki-based), JBehave, Cucumber (see later)

Acceptance testing vs. system testing

System tests ⇒ build the thing right (things should behave correctly)

Acceptance tests ⇒ build the right thing (validate what you want)


Scalability testing

Scalability testing: how is the ability of the system to accomodateincreasing loads?

Stress testing

Testing the system under unusual conditions (e.g., DoS attack)

Testing the upper limits (e.g., by exceeding the max load ordiminishing resources) until the SW crashes

Strictly related to reliability and availability requirements

Load testing

Testing system performance under high-load conditions

Example metrics: maximum/average response time, throughput

Tools: Gatlinga, The Grinderb,

ahttp://gatling.io/bhttp://grinder.sourceforge.net/


http://gatling.io/

http://grinder.sourceforge.net/

Code/Test coverage

Coverage

Several definitions/metrics (and inconsistent terminology)

Many “things” can be “covered”, and at many levels

Code coverage: which/how much code is executed during testingI Mainly used for finding untested codea

I Note: 100% coverage doesn’t mean that tests cover all the scenarios

Test coverage: how much of the behaviour is tested

The goal is not 100% coverage → testing the “right things”

ahttps://martinfowler.com/bliki/TestCoverage.html

Further readings“Coverage is not strongly correlated with test suite effectiveness” (Inozemtseva andHolmes 2014) [3]

“How to misuse code coverage” (Marick 1999) [5]

“A survey of coverage-based testing tools” (Yang, Li, and Weiss 2009) [9]


https://martinfowler.com/bliki/TestCoverage.html

Code coverage in practice

Configuration in sbt

project/plugins.sbt

1 addSbtPlugin("org.scoverage" % "sbt -scoverage" % "1.5.0")

Enable coverage in build.sbt

1 coverageEnabled := true

Usage

1 $ sbt

2 sbt > coverage // Enables coverage

3 sbt > test // Execute tests

4 sbt > coverageReport // Generate coverage reports

5 sbt > exit

6

7 $ open target/scala -2.11/ scoverage -report/index.html


Outline

1 Quick recap

2 sbt



5 Test doubles

6 More on testing


Testing in Scala: ScalaTest

Key features

Provides integration with many tools to provide a full, seamlesstesting experience

I JUnit/TestNG, ScalaCheck, JMock, ScalaMock, ...I Ant, Maven, sbtI Eclipse, NetBeans, IntelliJ

Supports several styles of testing out of the boxI Recommendation: 1 style for unit testing, 1 style for acceptance testing

Configuration in sbt:

1 val scalatest = "org.scalatest" % "scalatest_2 .12" % "3.0.1" % "

test"

2

3 val commonsettings = Seq(scalaVersion := "2.12.2")

4

5 lazy val root = (project in file(".")).

6 settings(commonsettings).

7 settings(name := "hello -scalatest",

8 libraryDependencies ++= Seq(scalatest))


ScalaTest: styles I

http://www.scalatest.org/user_guide/selecting_a_style

FunSuite: test functions as in classical XUnit libraries

1 class SetSuite extends FunSuite {

2 test("An empty Set should have size 0") {

3 assert(Set.empty.size == 0)

4 }

5

6 test("Invoking head on an empty Set should produce

NoSuchElementException") {

7 assertThrows[NoSuchElementException] {

8 Set.empty.head

9 }

10 }

11 }


http://www.scalatest.org/user_guide/selecting_a_style

ScalaTest: styles II

FlatSpec: flat specifications a la BDD

BDD (Behaviour-Driven Development) is a refinement of (A)TDDI From “tests” to “behaviour specifications”I Emphasis on communicationI Stylistic impact + tooling impact (cf., Cucumber @ next slide)

1 class SetSpec extends FlatSpec {

2 "An empty Set" should "have size 0" in {


4 }

5

6 it should "raise NoSuchElementException when head is invoked" in

{


8 Set.empty.head

9 }

10 }

11 }


ScalaTest: styles III

FunSpec: specifications a la BDD with (nested) describe blocks

1 class SetSpec extends FunSpec {

2 describe("A Set") {

3 describe("when empty") {

4 it("should have size 0") {


6 }

7

8 it("should raise NoSuchElementException for head") {


10 Set.empty.head

11 }

12 }

13 }

14 }

15 }


ScalaTest: styles IV

Many styles: recap

FunSuite: for xUnit people

FlatSpec: transition from xUnit to BDD-style

FunSpec: for BDD people

... other variants, e.g., giving more flexibility in the BDD style, ...

PropSpec: for property-based testing (see @ next slide)

FeatureSpec: for acceptance BDD (see @ next slide)

Guidelines for choice

One main style for unit testing (e.g., FlatSpec)

One main style for acceptance testing (e.g., FeatureSpec)

PropSpec when property-based testing suited


Outline

1 Quick recap

2 sbt



5 Test doubles

6 More on testing


Test doubles [6]: why and what

Test doubles: objects to be substituted for the real implementation fortesting purposes

Allow to gain full control if the context/environment of a piece ofcode

Test doubles are essential for good test automation

Allowing isolation of code under test (from its dependencies)

Speeding-up test execution

Making random behavior deterministic

Simulating particular conditions that would be difficult to create

Observing state & interaction otherwise invisible


Kinds of test doubles

Kinds of test doubles

Dummy: objects passed around but never actually used; often usedto fill parameter lists (e.g., as null)

Stubs (“unusually short things”): simple implementations, e.g., withmethods that do nothing or return default values

Fake objects: have working implementations but provide afast/simple alternative (e.g., in-memory DB)

Test spies: stubs that record information about how they are called

Mocks: test spies configured/prescribed to behave in a certain wayunder certain circumstances

I Tests also verify mocks are used in a certain way (white-box)

References

http://xunitpatterns.com/TestDouble.html

https://martinfowler.com/articles/mocksArentStubs.html


http://xunitpatterns.com/Test Double.html

https://martinfowler.com/articles/mocksArentStubs.html

Mocking with ScalaMock I

Usage

1 libraryDependencies += "org.scalamock" %% "scalamock -scalatest -support" % "3.5.0" % Test

ScalaMock can be used in tests written in ScalaTest (or Specs2)

In ScalaTest, mix trait MockFactory in your test class

Mocking styles in ScalaMock

Expectations-First style: expectations are set on mocks beforeexercising the System Under Test (SUT)

I If these expectations are not met, the test fails

Record-then-Verify style: expectations are verified on stubs afterthe SUT has executed

I In ScalaMock, this is supported by stubs

(1) for objects for which we have strict expectations, (2) otherwise


Mocking with ScalaMock II

SUT

1 case class Request(body: String)

2 case class Response(content: String)

34 class Server {

5 def processRequest(req: Request): Response = Response(req.body+ "’s Response")

6 def serve(req: Request): Response = processRequest(req)

7 }

89 trait Cache[K,T] { // Note: we provide no implementation class yet

10 def cached(key: K): Boolean

11 def get(key: K): T

12 def put(key: K, value: T): Unit

13 }

1415 class ServerWithCache(private val cache: Cache[Request , Response],

16 private val cacheable: Request => Boolean) extends Server {

17 override def serve(req: Request) = if(! cacheable(req)) super.serve(req) else {

18 if (cache.cached(req)) {

19 cache.get(req)

20 } else {

21 val res = processRequest(req)

22 cache.put(req , res)

23 res

24 }

25 }

26 }


Mocking with ScalaMock III

1 class TestWithMock extends FunSuite with MockFactory with Matchers {

2 test("Test server with cache"){

3 // Mocks/stubs for dependencies

4 val cacheMock = mock[Cache[Request ,Response ]]

5 val cacheableStub = stubFunction[Request , Boolean]

6 cacheableStub.when (*).returns(true)

78 // Wire SUT with stubbed/mocked dependencies

9 val server = new ServerWithCache(cacheMock , cacheableStub)

1011 // Arrange

12 val request = Request("Some request")

13 val expectedResponse = Response(request.body+"’s Response")

1415 // Mock expectations

16 inSequence {

17 (cacheMock.cached _).expects(request).returning(false)

18 (cacheMock.put _).expects(request , *)

19 (cacheMock.cached _).expects(request).returning(true)

20 (cacheMock.get _).expects(request).returning(expectedResponse)

21 }

2223 // Act + Assert

24 server.serve(request) shouldEqual expectedResponse

25 server.serve(request) shouldEqual expectedResponse

26 cacheableStub.verify(request).twice () // (cf., test spy)

27 }

28 }


Outline

1 Quick recap

2 sbt



5 Test doubles

6 More on testing


On effective unit testing

Key point: (software) quality should also be applied to testing code

3 key properties: readability, maintainability, trustworthiness

Good unit tests should

be automated and repeatable

be easy to implement

be relevant tomorrow

be runnable by anyone with ease

run quickly

be consistent in its results

have full control of the unit under test

be fully isolated

be communicative when failing


On testable design

Testability of software components

Refers to how much effort is needed for testingI Wrt a certain testing technique

Factors affecting testability includeI ObservabilityI ControllabilityI Number of dependencies and how they are resolvedI Number of responsibilities

Principles of testable design

Modularity

SOLID (SRP, OCP, LSP, ISP, DIP)

Inversion of ControlI Cf., Dependency Injection (note: not “Dependency Inversion”)


Property-based testing with ScalaCheck (i)

Property-based testing: key idea

Decoupling specification of behaviour from creation of test cases

1) You express properties that must hold for certain types of data

2) Tools automatically generate test cases trying to falsify those propsI When a prop is falsified, the (minimal) counterexample is given

ScalaCheck

Key abstractions

org.scalacheck.Prop: a property, the testable unit in ScalaCheckI Kinds: universally quantified, existentially quantified, conditional, ..I Combinators to compose props, e.g., p && q, atLeastOne(p,q,r)

org.scalacheck.Gen: a generator, for producing test dataI Combinators for creating custom generators


Property-based testing with ScalaCheck (ii)

libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.13.4" % "test"

1 import org.scalacheck .{Arbitrary , Prop , Properties}

2 import org.scalacheck.Prop.{ forAll , exists }

34 object TestOnNumbers extends Properties("Numbers") {

5 property("Sum is associative") = forAll{ (a: Int , b: Int , c: Int) =>

6 (a+b)+c == a+(b+c)

7 }

8 property("Sum is commutative") = forAll { (a: Int , b: Int) =>

9 a+b == b+a

10 }

11 property("Sum has zero as identity") = forAll { (a: Int) =>

12 a + 0 == a && 0 + a == a

13 }

14 property("Diff is not associative") = forAll{ (a: Int , b: Int) =>

15 exists{ c:Int => (a-b)-c != a-(b-c) }

16 }

17 }

1819 object TestOnLists extends Properties("Seqs") {

20 def genericSizeProp[A:Arbitrary ]: Prop = forAll{ (l1: Seq[A], l2: Seq[A]) =>

21 (l1++l2).size == l1.size + l2.size

22 }

23 property("Size of concatenation") = Prop.all(

24 genericSizeProp[Int], genericSizeProp[String], genericSizeProp [(Boolean ,Double)])

25 property("Reverse") = forAll { (l1: Seq[Int], l2: Seq[Int]) =>

26 l1.reverse.reverse == l1

27 }

28 }


ScalaTest: property-based testing support

PropSpec: for property-based test suites

Two styles for property checks:

1. ScalaTest style: mix PropertyChecks trait in

2. ScalaCheck style: mix Checkers trait in

1 class SetSpec extends PropSpec with PropertyChecks with Checkers with Matchers {

2 val examples = Table("set",BitSet.empty , HashSet.empty[Int], TreeSet.empty[Int])

3 val examplesGen = Gen.oneOf(BitSet.empty , HashSet.empty[Int], TreeSet.empty[Int])

45 property("an empty Set should have size 0") {

6 // ScalaTest ’s property -based testing style

7 forAll(examples) { set => set.size should be (0) }

8 // ScalaCheck ’s property -based testing style

9 check { Prop.forAll(examplesGen) { set => set.size == 0 } }

10 }

1112 property("invoking head on an empty set should produce NoSuchElementException") {

13 // ScalaTest ’s property -based testing style

14 forAll(examples) { set => a [NoSuchElementException] should be thrownBy{set.head} }

15 // ScalaCheck ’s property -based testing style

16 check { Prop.forAll(examplesGen) { set =>

17 Prop.throws(classOf[NoSuchElementException ])(set.head) }

18 }

19 }

20 }


Behaviour-Driven Development (BDD) with Cucumber (i)

Motivation

Remember: acceptance testing = are we building the right thing?

Agile approach: iterative, incremental development and frequentinteractions with stakeholders to get feedback

But requirements need to be communicated, different stakeholdersmay use their particular jargon, ..

Behaviour-Driven Development (BDD)

⇒ Idea: collaborative specification of acceptance tests

≈ Acceptance Test-Driven Dev. (ATDD) + customer-orientation

Acceptance tests as executable specs and a living documentationI Pros: ubiquitous language (cf., DDD) + “definitive source of truth”

Acceptance tests specified as examples of the way we want thesystem to behave in particular scenarios


Behaviour-Driven Development (BDD) with Cucumber (ii)

Cucumber: basics

Features are high-level specs expressed by a user-perspective in aEnglish-like DSL called Gherkin

Features consist of multiple scenarios, representing concreteexamples/use cases

Scenarios are structured into multiple steps (given, when, then)

sbt configuration

1 val commons = Seq(scalaVersion := "2.11.8")

2 val cuke_runner = "com.waioeka.sbt" %% "cucumber -runner" % "0.0.8"

3 val cuke_scala = "info.cukes" %% "cucumber -scala" % "1.2.4" % Test

4 val cuke_junit = "info.cukes" % "cucumber -junit" % "1.2.4" % Test

5

6 lazy val root = (project in file(".")).settings(commons).settings(

7 name := "hello -scala -cucumber",

8 libraryDependencies ++= Seq(cuke_runner ,cuke_scala ,cuke_junit))


Behaviour-Driven Development (BDD) with Cucumber (iii)

src/test/resources/features/MyFeature.feature

1 Feature: Numbers

23 Scenario: I want to multiply two nums

4 Given one operand 7

5 And another operand 8

6 When I multiply them together

7 Then I should obtain result 56

src/test/scala/RunCucumberTests.scala

1 @RunWith(classOf[Cucumber ])

2 @CucumberOptions(features = Array("classpath:features/MyFeature.feature"),

3 tags = Array("~@Wip"), glue = Array("classpath:steps"),

4 plugin = Array("pretty", "html:target/cucumber/html"))

5 class RunCucumberTests

src/test/scala/steps/CucumberSteps.scala

1 class CucumberSteps extends ScalaDsl with EN {

2 var (a, b, sum) = (0, 0, 0)

34 Given("""ône operand (\d+)$"""){ a = (_:Int) }

5 Given("""ânother operand (\d+)$"""){ b = (_:Int) }

6 When("""Î multiply them together$"""){ sum = a*b }

7 Then("""Î should obtain result (\d+)$"""){ (expectedSum:Int) =>

8 Assert.assertEquals(expectedSum , sum)

9 }

10 }


ScalaTest: acceptance testing style

FeatureSpec: for acceptance testing suites

1 class TVSet {

2 private var on: Boolean = false

3 def isOn: Boolean = on

4 def pressPowerButton () { on = !on }

5 }

67 class TVSetSpec extends FeatureSpec with GivenWhenThen {

8 info("As a TV set owner")

9 info("I want to be able to turn the TV on and off")

10 info("So I can watch TV when I want")

11 info("And save energy when I’m not watching TV")

1213 feature("TV power button") {

14 scenario("User presses power button when TV is off") {

15 Given("a TV set that is switched off")

16 val tv = new TVSet

17 assert (!tv.isOn)

1819 When("the power button is pressed")

20 tv.pressPowerButton ()

2122 Then("the TV should switch on")

23 assert(tv.isOn)

24 }

25 }

2627 scenario("User presses power button when TV is on") { ??? }

28 }


Web application testing with Selenium

Testing web applications

Selenium is a tool that provides support for browser-based tests

ScalaTest includes a DSL to drive Selenium tests

Ref.: http://www.scalatest.org/user_guide/using_selenium

Sbt dep: "org.seleniumhq.selenium" % "selenium-java" % "2.35.0" % "test"

1 class GoogletestSpec extends FlatSpec with Matchers with HtmlUnit {

2 "Google search" should "work" in {

3 go to "http ://www.google.com"

4 pageTitle should be ("Google")

5 executeScript("return document.title") shouldEqual pageTitle

6

7 click on "q" // By name of the field

8 textField("q").value = "selenium"

9 submit ()

10

11 eventually(timeout (2 seconds)) {

12 pageTitle should startWith ("selenium - ")

13 }

14 }

15 }R.Casadei (Universita di Bologna) Testing a.a. 2016/2017 41 / 47

http://www.scalatest.org/user_guide/using_selenium

Testing GUI-based programs

Key question

What does it mean to test a GUI-based program?

Answer “what should I test” before “how”

Approaches

1. Test monkey

2. Test monkey + logging

3. Mouse/Keyboard capture & replay toolsI Scripts

4. Programmatic access to GUI

5. Image recognition to identify/control GUI componentsI E.g.: SikuliXa

ahttp://sikulix.com/

https://en.wikipedia.org/wiki/Comparison_of_GUI_testing_tools


http://sikulix.com/

https://en.wikipedia.org/wiki/Comparison_of_GUI_testing_tools

Microbenchmarking with ScalaMeter (i)

A microbenchmark is a measurement of the running time of someisolated piece of codeIssues in doing microbenchmarking

Microbenchmark conditions/inputs are hard to reproduce

Performance measurement in general introduces observer bias

In general, runtime behaviour is nondeterministicI Inaccuracies in performance measurements can be due to

JIT-compilation, classloading, automatic memory management (GC), ..

Performance metrics (such as running time) inherently give anincomplete picture about the performance characteristics

However, microbenchmarking may still be useful, e.g.,

for verifying that an optimisation is an optimisation at all

for comparison of different implementations wrt specific conditions

for supporting performance regression testing


Microbenchmarking with ScalaMeter (ii)

1 object Mapper {

2 def map1[T](xs: List[T])(f: T=>T): List[T] = xs.map(f)

3 def map2[T](xs: List[T])(f: T=>T): List[T] = xs.reverse.reverse.map(f)

4 }

56 import org.scalameter.api._ // ... other imports

78 object RegressionTest extends Bench.OnlineRegressionReport {

9 override val persistor: Persistor = SerializationPersistor ()

10 override val reporter: Reporter[Double] = Reporter.Composite(

11 new RegressionReporter(tester ,historian), HtmlReporter (! online))

1213 val sizes = Gen.range("size")(1000, 10000 , 3000)

14 val lists = for (sz <- sizes) yield (0 until sz).toList

1516 performance of "List" in {

17 measure method "map" in {

18 using(lists) config (

19 exec.benchRuns -> 20, exec.independentSamples -> 1

20 ) in { xs =>

21 Mapper.map1(xs)(_+1)

22 }

23 }

24 }

25 }


References and further readings

Effective unit testing

Roy Osherove. The Art of Unit Testing: With Examples in .Net. 1st. Greenwich, CT,USA: Manning Publications Co., 2009. isbn: 1933988274, 9781933988276

L. Koskela. Effective Unit Testing: A Guide for Java Developers. Running Series.Manning, 2013. isbn: 9781935182573

Gerard Meszaros. XUnit Test Patterns: Refactoring Test Code. Upper Saddle River, NJ,USA: Prentice Hall PTR, 2006. isbn: 0131495054

On specific topics

Dorothy Graham and Mark Fewster. Experiences of Test Automation: Case Studies ofSoftware Test Automation. 1st. Addison-Wesley Professional, 2012. isbn: 0321754069,9780321754066

Max Guernsey III. Test-Driven Database Development: Unlocking Agility. 1st.Addison-Wesley Professional, 2013. isbn: 032178412X, 9780321784124

S. Rose, M. Wynne, and A. Hellesoy. The Cucumber for Java Book: Behaviour-DrivenDevelopment for Testers and Developers. Pragmatic programmers. Pragmatic Bookshelf,2015. isbn: 9781941222294


Bibliography I

Dorothy Graham and Mark Fewster. Experiences of Test Automation: CaseStudies of Software Test Automation. 1st. Addison-Wesley Professional, 2012.isbn: 0321754069, 9780321754066.

Max Guernsey III. Test-Driven Database Development: Unlocking Agility. 1st.Addison-Wesley Professional, 2013. isbn: 032178412X, 9780321784124.

Laura Inozemtseva and Reid Holmes. “Coverage is not strongly correlated withtest suite effectiveness”. In: Proceedings of the 36th International Conference onSoftware Engineering. ACM. 2014, pp. 435–445.

L. Koskela. Effective Unit Testing: A Guide for Java Developers. Running Series.Manning, 2013. isbn: 9781935182573.

Brian Marick et al. “How to misuse code coverage”. In: Proceedings of the 16thInterational Conference on Testing Computer Software. 1999, pp. 16–18.

Gerard Meszaros. XUnit Test Patterns: Refactoring Test Code. Upper SaddleRiver, NJ, USA: Prentice Hall PTR, 2006. isbn: 0131495054.


Bibliography II

Roy Osherove. The Art of Unit Testing: With Examples in .Net. 1st. Greenwich,CT, USA: Manning Publications Co., 2009. isbn: 1933988274, 9781933988276.

S. Rose, M. Wynne, and A. Hellesoy. The Cucumber for Java Book:Behaviour-Driven Development for Testers and Developers. Pragmaticprogrammers. Pragmatic Bookshelf, 2015. isbn: 9781941222294.

Qian Yang, J Jenny Li, and David M Weiss. “A survey of coverage-based testingtools”. In: The Computer Journal 52.5 (2009), pp. 589–597.
