mutants, tests and zombies - testconf.ru...mutation testing can be used to evaluate the quality of...
TRANSCRIPT
![Page 1: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/1.jpg)
Mutants, tests and zombies
Alexander Todorovhttp://atodorov.org
@atodorov_TestCon Moscow 2018
![Page 2: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/2.jpg)
S
![Page 3: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/3.jpg)
![Page 4: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/4.jpg)
![Page 5: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/5.jpg)
Is my test suite good enough
?
![Page 6: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/6.jpg)
What is a mutation ?
![Page 7: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/7.jpg)
- if age > 18 then
+ if age < 18 then
buy(beer)
![Page 8: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/8.jpg)
Федеральный закон о государственном регулировании
производства и оборота этилового спирта, алкогольной
и спиртосодержащей продукции says 18+
![Page 9: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/9.jpg)
- had_lunch = True
+ had_lunch = False
![Page 10: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/10.jpg)
THAT MOMENT WHEN
YOU'RE GRUMPY AS H*LL BECAUSE YOUDIDN'T HAVE LUNCH
![Page 11: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/11.jpg)
while True do
- break
+ continue
![Page 12: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/12.jpg)
![Page 13: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/13.jpg)
Definition
A mutation is a controlled change in the SUT
which produces a valid syntax but changes
program behavior!
Mutation testing can be used to evaluate the
quality of existing test suites!
It can find missing or inadequate tests or dead
code!
![Page 14: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/14.jpg)
Algorithm for mutation testing
for operator in mutation-operators:
for site in operator.sites(code):
for mutant in operator.mutate(site):
run_tests(mutant)
https://github.com/sixty-north/cosmic-ray
![Page 15: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/15.jpg)
3 rules to kill mutants !
run_tests(vanilla_code) MUST PASS
if run_tests(mutated_code) == FAIL:
mutant dies
if run_tests(mutated_code) == PASS:
zombie
![Page 16: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/16.jpg)
![Page 17: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/17.jpg)
class Desk < ApplicationRecord
... skip ...
def upcase_language_code
language_code&.upcase
end
end
![Page 18: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/18.jpg)
@@ -1,4 +1,4 @@
def upcase_language_code
- language_code&.upcase
+ nil
end
![Page 19: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/19.jpg)
еxpect(
desk.upcase_language_code
).to_not be_nil
![Page 20: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/20.jpg)
@@ -1,4 +1,4 @@
def upcase_language_code
- language_code&.upcase
+ self
end
![Page 21: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/21.jpg)
еxpect(
desk.upcase_language_code
).to be_instance_of(String)
![Page 22: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/22.jpg)
@@ -1,4 +1,4 @@
def upcase_language_code
- language_code&.upcase
+ language_code
end
![Page 23: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/23.jpg)
desk.language_code = 'ru'
expect(
desk.upcase_language_code
).to eq('RU')
![Page 24: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/24.jpg)
@@ -1,4 +1,4 @@
def upcase_language_code
- language_code&.upcase
+ language_code.upcase
end
![Page 25: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/25.jpg)
desk.language_code = nil
expect(
desk.upcase_language_code
).to be_nil
![Page 26: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/26.jpg)
Is my test suite good enough
?
![Page 27: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/27.jpg)
![Page 28: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/28.jpg)
http://kaner.com/pdfs/negligence_and_testing_coverage.pdf
![Page 29: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/29.jpg)
Mutation testing vs. coverage
●http://bit.ly/GTAC2015mutation
●http://bit.ly/GTAC2016coverage
●http://bit.ly/MutationVsCoverage
![Page 30: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/30.jpg)
SUT: pelican-ab, v0.2.1
AB_EXPERIMENT="control" make github
AB_EXPERIMENT="123" make github
AB_EXPERIMENT="xy" make github
●100% branch & mutation coverage
●Bug when
DELETE_OUTPUT_DIRECTORY=True
![Page 31: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/31.jpg)
SUT: pelican-ab, v0.2.1
●Recursion loop bug when class is inherited
- super(self.__class__, self).....
+ super(PelicanAbExperimentWriter, self)....
●See Pylint #1109 for more info
![Page 32: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/32.jpg)
How slow is it ?
https://github.com/rhinstaller/pykickstart/
●Full mutation test run: > 6 days
●After divide and conquer & fail fast: 6.5 hrs
![Page 33: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/33.jpg)
When to use
mutation testing ?
![Page 34: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/34.jpg)
Mutation testing tools
●Python – sixty-north/cosmic-ray
●Ruby – mbj/mutant
●Java – hcoles/pitest
●JavaScript - stryker-mutator/stryker
●PHP - padraic/humbug
●LLVM based: C, C++, Rust, etc – mull-
project/mull
https://mutation-testing.slack.com/
![Page 35: Mutants, tests and zombies - testconf.ru...Mutation testing can be used to evaluate the quality of existing test suites! It can find missing or inadequate tests or dead code! Algorithm](https://reader036.vdocuments.mx/reader036/viewer/2022071216/6046ed5230013e665e6472d5/html5/thumbnails/35.jpg)
СПАСИБО