writing acceptable patches: an empirical study of open source project patches
DESCRIPTION
Presented at ICSME2014 http://www.icsme.org/TRANSCRIPT
Writing Acceptable Patches: An Empirical Study of Open Source Project Patches Yida Tao*, DongGyun Han§ and Sunghun Kim**The Hong Kong University of Science and Technology§KAIST Institute for IT Convergence, Korea Advanced Institute of Science and Technology
Developers submit many patches…but not all of them can be accepted
Code Reviewer
2
Developers submit many patches…but not all of them can be accepted
Patch Reviewer
“IT’S A NO FROM ME!”
3
0%
10%
20%
30%
40%
50%
60%
70%
Firefox(Baysal et al. WCRE’12)
Apache(Rigby and German, 2006)
FLAC,OpenAFS(Weißgerber et al. MSR’08)
Patch rejection & resubmission
56%
40%
60%
4
Patch rejection & resubmission = ? increased interruption and workload
5
Patch rejection & resubmission = increased interruption and workload
6
We hope our patches to be accepted right after the first or early rounds of code review
7
How to write acceptable patches?
8
How to write acceptable patches?
• Learn from “past failures”• Q1: Why are patches rejected?
•Understand the priority• Q2: Which reasons are more decisive to reject a patch?• Q3: Which reasons are more difficult to judge?
•Analyze primary stakeholders• Q4: Do patch reviewers and writers have unanimous
patch evaluation criteria?
9
How to write acceptable patches?
• Learn from “past failures”• Q1: Why are patches rejected?
•Understand the priority• Q2: Which reasons are more decisive to reject a patch?• Q3: Which reasons are more difficult to judge?
•Analyze primary stakeholders• Q4: Do patch reviewers and writers have unanimous
patch evaluation criteria?
10
How to write acceptable patches?
• Learn from “past failures”• Q1: Why are patches rejected?
•Understand the priority• Q2: Which reasons are more decisive to reject a patch?• Q3: Which reasons are more difficult to judge?
•Analyze primary stakeholders• Q4: Do patch reviewers and writers have unanimous
patch evaluation criteria?
11
• Learn from “past failures”• Q1: Why are patches rejected?
•Understand the priority• Q2: Which reasons are more decisive to reject a patch?• Q3: Which reasons are more difficult to judge?
•Analyze primary stakeholders• Q4: Do patch reviewers and writers have unanimous
patch evaluation criteria?
Patch extraction & inspection
Developer survey
Literature12
Patch Extraction & Inspection
13
Extracting Rejected Patches
• Projects• Eclipse and Mozilla
• Bug tracking system• Bugzilla
• Criterion for rejected patch• Patches with explicit review- flag
• Number of patches• 300 randomly inspected
• 162 Eclipse + 138 Mozilla
Patch attachments for Eclipse bug#233156
spell checking performance test (patch)2009-11-23
no flags
performance test (5.69 KB, patch)2009-11-30
no flags
performance test + fix (12.95 KB, patch)2009-12-01
review-
fix (7.63 KB, patch)2010-01-04
review-
performance test (6.53KB, patch)2010-01-04
review+
fix (10.08 KB, patch)2010-01-14
review+
14
Patch Inspection
•Open coding• Review comments of 300 rejected patches
• # open codes: 637
“The patch fixes CCE but introduces a new bug: … Also the parameter should be renamed to …“
Open code:• new bug introduced• bad parameter name
15
Patch Inspection
Open code:“malformed patch”
Open code: “wrong indentation”
Theme:Violating coding styles
16
•Open coding• Review comments of
300 rejected patches
• # open codes: 637
•Axial coding
Patch Inspection
Open code:“malformed patch”
Open code: “wrong indentation”
Theme:Violating coding styles
17
•Open coding• Review comments of
300 rejected patches
• # open codes: 637
•Axial coding• 12 themes why patches
are rejected
Theme Frequency
24.6%
18
Suboptimal solution
Theme Frequency
24.6%
19
Incomplete fix22.3%
Theme Frequency
24.6%
20
22.3%
Including unnecessary changes11.9%
Violating coding styles9.7%
Theme Frequency
24.6%
21
22.3%
11.9%
Bad Naming
Inconsistent, misleadingdocumentationMissing documentation
Test Failure
Introducing New Bugs
Compilation errors
Duplicate code
6.3%
6.1%
5.7%
5.5%
Theme Frequency
22
24.6%
22.3%
11.9%
9.7%
Suboptimal solution, 24.6%
Incomplete fix, 22.3%
Including unnecessary
change, 11.9%Violating coding style guidelines,
9.7%
Bad naming, 6.3%
Inconsistent, misleading
documentation, 6.1%
Missing documentation,
5.7%
Test failures, 5.5%
Introducing new bugs, 4.1%
Compilation errors, 1.9%Duplication, 1.6%
Patch size too large, 0.3%
Developer survey12 patch rejection reasons from manual inspection
• Decisive score of each reason• Difficulty to judge each reason246 / 1785 (14%)
Eclipse and Mozilla developers
23
Q2: How decisive each rejection reason is?
24
Patch reviewers have very low tolerance for patches that fail compilation or tests, or introduce new bugs
25
Inconsistent or misleading documentation is highly unacceptable and way worse than missing documentation
26
Inconsistent or misleading documentation is highly unacceptable and way worse than missing documentation
27
Reviewers reject a large patch not solely because of its size, but mainly because the underlying reasons that induce its large size
28
Reviewers reject a large patch not solely because of its size, but mainly because the underlying reasons that induce its large size
29
Q3: Which patch-rejection reasons are more difficult to judge?
30
Ver
y d
iffi
cult
to
jud
geV
ery
easy
Not decisive Very decisive to reject a patch 31
Test failures
Compile errors
Inconsistent, misleading documentation
Ver
y d
iffi
cult
to
jud
geV
ery
easy
Not decisive Very decisive to reject a patch 32
Ver
y d
iffi
cult
to
jud
geV
ery
easy
Not decisive Very decisive to reject a patch
Unnecessary changes
Duplication
33
Ver
y d
iffi
cult
to
jud
geV
ery
easy
Not decisive Very decisive to reject a patch
Incomplete fixSuboptimal solution
Introduce new bugs
34
Q4: Do patch reviewers and writers have unanimous patch evaluation criteria?
35
Distinguish patch reviewers and patch writers
•Method• Patch writer: write ≥ 1 patch• Patch reviewer: review ≥ 1 patch or both
•Responses• 246 respondents• 131 patch writers, 115 patch reviewers
36
37
2 3 4 5
Compilation errors
Test Failures
Introducing new bugs
Inconsistent Documentation
Suboptimal solution
Incomplete fix
Duplication
Unnecessary changes
Violating coding styles
Bad naming
Missing documentation
Patch size too large
Patch reviewer Patch writer
Avg. Decisive Score
38
2 3 4 5
Compilation errors
Test Failures
Introducing new bugs
Inconsistent Documentation
Suboptimal solution
Incomplete fix
Duplication
Unnecessary changes
Violating coding styles
Bad naming
Missing documentation
Patch size too large
Patch reviewer Patch writer
Avg. Decisive Score
39
2 3 4 5
Compilation errors
Test Failures
Introducing new bugs
Inconsistent Documentation
Suboptimal solution
Incomplete fix
Duplication
Unnecessary changes
Violating coding styles
Bad naming
Missing documentation
Patch size too large
Patch reviewer Patch writer
Avg. Decisive Score
Patch reviewers tend to have stricter criteria on patch quality compared to patch writers
Why are patches rejected?
40
Why are patches rejected?
Patch inspection
41
Why are patches rejected?
Patch inspection Developer survey
+
42
Why are patches rejected?
Patch inspection Developer survey Literature
+ +
43
30 patch rejection reasons in 5 categories
Why are patches rejected?
Patch inspection Developer survey Literature
+ +
44
30 patch rejection reasons in 5 categories
1. Problematic implementation or solution
2. Difficult to read or maintain
3. Lack of communication or trust
4. Deviating from the project focus or scope
5. Affecting the development schedule
45
• Compilation & test failures
• Incomplete
• Suboptimal
• Solution too aggressive or hostile for end users
46
1. Problematic implementation or solution
47
2. Difficult to read or maintain
• Coding style violation
• Inconsistent documentation
• Using deprecated API
• No accompanied test cases
3. Lack of communication or trust• No discussion before patch submission• Patch writers are not responsive
4. Deviating from the project scope or focus• Irrelevant or obsolete
5. Affecting the development schedule• Freeze the progress of other features
48
Please check our paper for more details on patch rejection reasons
49
Summary
Writing acceptable patches
50
Checklist
* Compile error* Test failure* Documentation* Duplication* Unnecessary changes
51
Checklist
* Compile error* Test failure* Documentation* Duplication* Unnecessary changes
52
Communication
* Discussion prior to patch submission
* Timely response to review comments
Checklist
* Compile error* Test failure* Documentation* Duplication* Unnecessary changes
53
Communication
* Discussion prior to patch submission
* Timely response to review comments
Call for industrial &research efforts
* Introduce new bugs
* Suboptimal solutions
* Incomplete fix