analysing test results over time listen beyond the pass/fail · analysing test results over time...
TRANSCRIPT
![Page 1: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/1.jpg)
Listen beyond the pass/fail
Analysing test results over time
Lena Wiberg@LenaPejgan
![Page 2: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/2.jpg)
Lena Wiberg
SCI-FI/Fantasy & Games
Patterns & trends
Visual models
Dev. => Tester => Manager
![Page 3: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/3.jpg)
Awesome new job!
![Page 4: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/4.jpg)
Vision! Reality…
@LenaPejgan
![Page 5: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/5.jpg)
It works fine when I run it locally
@LenaPejgan
![Page 6: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/6.jpg)
The test environment was locked by another
user
@LenaPejgan
![Page 7: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/7.jpg)
We hit a random duality check
@LenaPejgan
![Page 8: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/8.jpg)
It was just an unexpected popup
@LenaPejgan
![Page 9: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/9.jpg)
System/service X had the wrong data/version
@LenaPejgan
![Page 10: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/10.jpg)
Oh, it’s just because of bad data
@LenaPejgan
![Page 11: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/11.jpg)
Run
Analyze
Setup
Fix @LenaPejgan
![Page 12: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/12.jpg)
Identify the problem
@LenaPejgan
![Page 13: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/13.jpg)
NOISE!
@LenaPejgan
![Page 14: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/14.jpg)
@LenaPejgan
FALSE SECURITY
![Page 15: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/15.jpg)
Feedback
Stability
Planning
Investment Is it stable enough to explore?
Did my changes break something?
Is this version ready for release?
Are we spending out money right?
@LenaPejgan
![Page 16: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/16.jpg)
It works fine when I run it
locally
Oh, it’s just because of bad data
The test environment was locked by another user
It was just an unexpected
popup
We hit a random
duality check
The integrating system had the
wrong backup/version
+@LenaPejgan
![Page 17: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/17.jpg)
Operations & Hardware
@LenaPejgan
”Our firewall doesn’t allow that”
”Oh, that’s when we do backups”
”We do patches every 3rd Monday”
![Page 18: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/18.jpg)
Test environments & Test data
@LenaPejgan
”The data was used up by another test”
”Someone else was using the environment”
”Service X was down/had the wrong data”
![Page 19: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/19.jpg)
Processes and communication
@LenaPejgan
”We didn’t have time to update the tests”
”We can only test version X”
”We didn’t know the code has changed”
![Page 20: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/20.jpg)
FIX!…Or an ACTUAL bug!
@LenaPejgan
TEAM AWESOME
![Page 21: Analysing test results over time Listen beyond the pass/fail · Analysing test results over time Lena Wiberg @LenaPejgan. Lena Wiberg SCI-FI/Fantasy & Games Patterns & trends Visual](https://reader034.vdocuments.mx/reader034/viewer/2022042309/5ed5860329851c5413653859/html5/thumbnails/21.jpg)
Summary•Don’t get (too) caught up in the now •Look past the obvious •Kon Mari those tests! Do they spark joy?•Don’t assume flaky can’t be fixed•Reuse is awesome. Except when it’s not…
@LenaPejgan