![Page 1: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/1.jpg)
Using (Bio)Metrics to Predict Code Quality Online
Sebastian Müller & Thomas FritzUniversity of Zurich
![Page 2: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/2.jpg)
“Every minute spent on not-quite-right code counts as interest on that debt.”
Ward Cunningham
1
![Page 3: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/3.jpg)
Detecting Quality Concerns
Code Reviews
Automatic approaches to detect quality concerns
2
![Page 4: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/4.jpg)
Detecting Quality Concerns
Code Reviews
Automatic approaches to detect quality concerns
Required metrics can often only be collected after change
task is completed
Do not take the individual differences between
developers into account
Time-consuming and require a lot of effort
2
![Page 5: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/5.jpg)
Biometric Sensing to DetectQuality Concerns
3
![Page 6: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/6.jpg)
Biometric Sensing to DetectQuality Concerns
3
![Page 7: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/7.jpg)
Biometric Sensing to DetectQuality Concerns
3
![Page 8: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/8.jpg)
Biometric Sensing to DetectQuality Concerns
3
![Page 9: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/9.jpg)
Biometric Sensing to DetectQuality Concerns
3
![Page 10: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/10.jpg)
Biometric Sensing to DetectQuality Concerns
Cognitive / emotional state Psychological aspectCognitive load Pupil sizeEmotion (Valence / Arousal) Eye blink rate
3
![Page 11: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/11.jpg)
EDA HR …HRV
Biometric measurements
Cognitive Load
Biometrics, Cognitive Load and Difficulty/Errors
4
![Page 12: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/12.jpg)
EDA HR …HRV
Biometric measurements
Cognitive Load
Task
Developer
task format & complexity, time pressure,
instructions, etc.
age, expertise, personality traits, etc.
Biometrics, Cognitive Load and Difficulty/Errors
4
![Page 13: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/13.jpg)
EDA HR …HRV
Biometric measurements
Cognitive Load
Task
Developer
task format & complexity, time pressure,
instructions, etc.
age, expertise, personality traits, etc.
Quality Concerns / Errors
Difficulty
Biometrics, Cognitive Load and Difficulty/Errors
4
![Page 14: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/14.jpg)
Research Questions
5
![Page 15: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/15.jpg)
Research Questions
Can biometrics be used to identify places in the code that are perceived to be more difficult by developers?1
5
![Page 16: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/16.jpg)
Research Questions
Can biometrics be used to identify places in the code that are perceived to be more difficult by developers?1Can we use biometrics to identify code quality concerns found through peer code reviews?2
5
![Page 17: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/17.jpg)
Research Questions
Can biometrics be used to identify places in the code that are perceived to be more difficult by developers?1Can we use biometrics to identify code quality concerns found through peer code reviews?2How do biometrics compare to more traditional metrics for detecting quality concerns?3
5
![Page 18: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/18.jpg)
Study Method
10 professional developers
Work as usual, on average 11.6 days
1 or 2 biometric sensors(chest & wrist band)
Code difficulty ratingsResults of peer code reviews
Code metrics: McCabe’s, Halstead’s, Fanout, …Interaction metrics: # edits, # selects, # edits / # selectsChange metrics: # lines added / removed
6
![Page 19: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/19.jpg)
Collected Data
116 developer work days
162 quality concerns in 1109 code elements(46 methods, 116 classes)
Perceived difficulty for 1480 classesPerceived difficulty for 1511 methods
~ 41 million biometric data points
7
![Page 20: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/20.jpg)
Research Approach
0 50 100 150 200 250 300 3502.85
2.9
2.95
3
3.05
3.1
3.15
3.2
3.25
* **
0 50 100 150 200 250 300 350-600
-400
-200
0
200
400
600
Developers’ perceived difficulty & quality concerns
Data recording
Data cleaning (e.g. noise canceling, filtering invalid data)
Feature extraction (e.g. normalization with baseline,
calculation of features)
Machine learning (e.g. labelling, splitting, classification)
Breathing & HR chestband
EDA & HR wristband
Breathing data
Phasic EDA
8
![Page 21: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/21.jpg)
Research Approach
0 50 100 150 200 250 300 3502.85
2.9
2.95
3
3.05
3.1
3.15
3.2
3.25
* **
0 50 100 150 200 250 300 350-600
-400
-200
0
200
400
600
Developers’ perceived difficulty & quality concerns
Data recording
Data cleaning (e.g. noise canceling, filtering invalid data)
Feature extraction (e.g. normalization with baseline,
calculation of features)
Machine learning (e.g. labelling, splitting, classification)
Breathing & HR chestband
EDA & HR wristband
Breathing data
Phasic EDA
8
![Page 22: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/22.jpg)
Research Approach
0 50 100 150 200 250 300 3502.85
2.9
2.95
3
3.05
3.1
3.15
3.2
3.25
* **
0 50 100 150 200 250 300 350-600
-400
-200
0
200
400
600
Developers’ perceived difficulty & quality concerns
Data recording
Data cleaning (e.g. noise canceling, filtering invalid data)
Feature extraction (e.g. normalization with baseline,
calculation of features)
Machine learning (e.g. labelling, splitting, classification)
Breathing & HR chestband
EDA & HR wristband
Breathing data
Phasic EDA
8
![Page 23: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/23.jpg)
Research Approach
Developers’ perceived difficulty & quality concerns
Data recording
Data cleaning (e.g. noise canceling, filtering invalid data)
Feature extraction (e.g. normalization with baseline,
calculation of features)
Machine learning (e.g. labelling, splitting, classification)
8
![Page 24: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/24.jpg)
Research Approach
Developers’ perceived difficulty & quality concerns
Data recording
Data cleaning (e.g. noise canceling, filtering invalid data)
Feature extraction (e.g. normalization with baseline,
calculation of features)
Machine learning (e.g. labelling, splitting, classification)
Data CleaningBiometric data is notoriously noisy
Applied noise cleaning techniques
9
![Page 25: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/25.jpg)
Research Approach
Developers’ perceived difficulty & quality concerns
Data recording
Data cleaning (e.g. noise canceling, filtering invalid data)
Feature extraction (e.g. normalization with baseline,
calculation of features)
Machine learning (e.g. labelling, splitting, classification)
Feature ExtractionFeature extraction following established methods
{Min, Max}PeakAmpl; ∆NumPhasicPeaks/Min, …EDA
Skin temperatureMeanTemp; ∆MeanTemp, …
HR(V)∆MeanHR; ∆VarianceHR, …
RR∆MeanRR; ∆Log10VarianceRR, …
9
![Page 26: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/26.jpg)
Research Approach
Developers’ perceived difficulty & quality concerns
Data recording
Data cleaning (e.g. noise canceling, filtering invalid data)
Feature extraction (e.g. normalization with baseline,
calculation of features)
Machine learning (e.g. labelling, splitting, classification)
Data Labelling and SplittingAssign difficulty ratings to code elements
Segment biometric data
9
![Page 27: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/27.jpg)
Data Labelling
time1 2 3 4 5 6
10
![Page 28: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/28.jpg)
Data Labelling
time
Mean
Heart
Rate
1 2 3 4 5 6
89 87
80
105 106110
beats
per min
10
![Page 29: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/29.jpg)
Data Labelling
time
Mean
Heart
Rate
1 2 3 4 5 6
Rating 1 Rating 2 Rating 3
89 87
80
105 106110
beats
per min
10
![Page 30: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/30.jpg)
Data Labelling
time
Mean
Heart
Rate
1 2 3 4 5 6
Rating 1 Rating 2 Rating 3
89 87
80
105 106110
beats
per min
10
![Page 31: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/31.jpg)
Data Labelling
time
Perc
eiv
ed
Difficulty
Mean
Heart
Rate
1 5
1 2 3 4 5 6
Rating 1 Rating 2 Rating 3
89 87
80
105 106110
beats
per min
3 1 5 5
10
![Page 32: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/32.jpg)
Research Approach
Developers’ perceived difficulty & quality concerns
Data recording
Data cleaning (e.g. noise canceling, filtering invalid data)
Feature extraction (e.g. normalization with baseline,
calculation of features)
Machine learning (e.g. labelling, splitting, classification)
Data Labelling and SplittingAssign difficulty ratings to code elements
Segment biometric data
11
![Page 33: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/33.jpg)
Research Approach
Developers’ perceived difficulty & quality concerns
Data recording
Data cleaning (e.g. noise canceling, filtering invalid data)
Feature extraction (e.g. normalization with baseline,
calculation of features)
Machine learning (e.g. labelling, splitting, classification)
Data Labelling and SplittingAssign difficulty ratings to code elements
Segment biometric data
Machine Learning ClassificationLeave-one-out approach with Random Forest
One classifier per metric and one for all
CE 1 CE 2 CE n-2 CE n-1 CE n
P01
…
11
![Page 34: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/34.jpg)
Results
![Page 35: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/35.jpg)
Developers’ Perceived Difficultyeasy
2073 (69.3%)
medium
829 (27.7%)
difficult
89 (3.0%)
Only few code elements are difficultDifficulty perception changes over time,
despite code metrics do not change13
![Page 36: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/36.jpg)
Predicting Perceived Difficulty (@ Commit Time)
14
![Page 37: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/37.jpg)
Predicting Perceived Difficulty (@ Commit Time)
14
![Page 38: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/38.jpg)
Predicting Perceived Difficulty (@ Commit Time)
14
![Page 39: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/39.jpg)
Predicting Perceived Difficulty(During Work)
15
![Page 40: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/40.jpg)
Predicting Perceived Difficulty(During Work)
15
![Page 41: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/41.jpg)
Predicting Perceived Difficulty(During Work)
15
![Page 42: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/42.jpg)
Predicting Perceived Difficulty(During Work)
Biometrics outperform traditional metrics in 3 out of 4 cases15
![Page 43: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/43.jpg)
Quality Concern Prediction
Of 580 reviewed classes, 95 had quality concerns
16
![Page 44: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/44.jpg)
Quality Concern Prediction
MetricQuality Concern no Quality Concern
Precision Recall Precision RecallAll 18 23 84 79
Biometric 22 40 86 72Code 17 30 84 72
Interaction 20 17 84 87Change 17 19 84 82
Of 580 reviewed classes, 95 had quality concerns
16
![Page 45: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/45.jpg)
Quality Concern Prediction
MetricQuality Concern no Quality Concern
Precision Recall Precision RecallAll 18 23 84 79
Biometric 22 40 86 72Code 17 30 84 72
Interaction 20 17 84 87Change 17 19 84 82
Of 580 reviewed classes, 95 had quality concerns
Biometric classifier outperforms all others
16
![Page 46: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/46.jpg)
Replication Study ShowsSimilar Results
5 professional developers
Work as usual,on average 5 days
1 or 2 biometric sensors(chest & wrist band)
780 difficulty ratings,but no quality concerns
Study Setup
Same metrics, except for change metrics
Results
Initial evidence that some findings can be replicated
Biometrics sometimes outperformed by code metric classifier
Many potential reasons for the differences in findings
17
![Page 47: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/47.jpg)
Contributions and Outlook
Two-week field study with professional developersBiometrics can identify difficulties and quality concerns
Biometrics outperform more traditional metrics
18
![Page 48: Using (Bio)Metrics To Predict Code Quality Online](https://reader034.vdocuments.mx/reader034/viewer/2022051503/5a6e08a87f8b9a505f8b4f81/html5/thumbnails/48.jpg)
Contributions and Outlook
Two-week field study with professional developers
New opportunities for developer support
Biometrics can identify difficulties and quality concerns
Biometrics outperform more traditional metrics
Support / intervene when developers experience difficulties
Identify quality concerns early on
Use new sensors to collect better data even less invasively
18