nasir eisty assistant professor testing and code review
TRANSCRIPT
Testing and Code Review Practices in Research
Software
Nasir EistyAssistant ProfessorDept. of CSSECal Poly State UniversitySan Luis Obispo, California
2
Who Am I?
Metric Process
Research Software
3
Software Quality
4
• Software Testing
Part #1
• Peer Code Review
Part #2
Contents
5
Recommendation
Part #1
Testing Research Software: A survey
6
Online Survey
7
Part #1 (Testing) Outline
• Demographics
• Level of knowledge research software developers have on testing
• Current testing practices in research software community
• Difficulties to test research software
• Compatibility of Commercial/IT testing techniques
• Improvement of the testing process in research software
8
Part #1 (Testing) Outline
• Demographics
• Level of knowledge research software developers have on testing
• Current testing practices in research software community
• Difficulties to test research software
• Compatibility of Commercial/IT testing techniques
• Improvement of the testing process in researach software
9
020406080
100120
Roles
10
Coun
t
0
10
20
30
40
50
60
70
Less than 1 year 1 year to lessthan 5 years
5 years to 10years
More than 10years
Years Worked
11
Coun
t
0102030405060708090
Project Stage
12
Coun
t
01020304050607080
1 to 5 6 to 10 11 to 15 16 to 20 More than 20
# Developers
13
Coun
t
Part #1 (Testing) Outline
• Demographics
• Level of knowledge research software developers have on testing
• Current testing practices in research software community
• Difficulties to test research software
• Compatibility of Commercial/IT testing techniques
• Improvement of the testing process in research software
14
0
10
20
30
40
50
60
Low Average High Very High
Knowledge of Testing
15
Coun
t
0
10
20
30
40
50
60
Very Low Low Average High Very High
Understanding Testing Concepts USED
16
Coun
t
0
10
20
30
40
50
60
70
Very Low Low Average High Very High
Understanding Testing Concepts NEEDED
17
Coun
t
Part #1 (Testing) Outline
• Demographics
• Level of knowledge research software developers have on testing
• Current testing practices in research software community
• Difficulties to test research software
• Compatibility of Commercial/IT testing techniques
• Improvement of the testing process in research software
18
0
10
20
30
40
50
60
Level 0 Level 1 Level 2 Level 3 Level 4
Testing Goals
19
Level 4 – Testing is a mental discipline that helps all researchers develop higher quality softwareLevel 1 – The Purpose of testing is to show correctnessLevel 3 – The Purpose of testing is not to prove anything specific, but to reduce the risk of using the software
Coun
t
020406080
100120140160
Unit Integration System Acceptance Module
Testing Methods Used
20
Coun
t
0
10
20
30
40
50
60
70
Always Most of the time Sometimes Rarely
Usefulness
21
Coun
t
Part #1 (Testing) Outline
• Demographics
• Level of knowledge research software developers have on testing
• Current testing practices in research software community
• Difficulties to test research software
• Compatibility of Commercial/IT testing techniques
• Improvement of the testing process in research software
22
0
10
20
30
40
50
60
Not Complex SlightlyComplex
ModeratelyComplex
VeryComplex
ExtremelyComplex
Complexity to Test
23
Coun
t
0 5 10 15 20 25 30
OtherCost
Legacy codeCodebase
Comparing with realityAffects CI
CultureSlow
Lack of knowledgeExternal dependencies
Lack of resourcesTest case design
Challenges
24Count
Part #1 (Testing) Outline
• Demographics
• Level of knowledge research software developers have on testing
• Current testing practices in research software community
• Difficulties to test research software
• Compatibility of Commercial/IT testing techniques
• Improvement of the testing process in research software
25
0
5
10
15
20
25
30
35
Always Most of thetime
Sometimes Rarely Never
Commercial/IT Testing Methods - Team
26
Coun
t
05
10152025303540
Always Most of thetime
Sometimes Rarely Never
Commercial/IT Testing Methods- Individual
27
Coun
t
05
1015202530354045
Very High High Average Low Very Low
Value Seen in Comm/IT Testing Methods
28
Coun
t
0 5 10 15 20
OtherRuntime restrictions
Difficult to useCost
InfrastructureLack of knowledge
MindsetLack of resources
Not useful
Challenges to Adapt Comm/IT Methods
29Count
0 2 4 6 8 10
Other
Benchmarks
CI issues
Slow
Lack of expertise
Challenges Not Met by Comm/IT Methods
30Count
Part #1 (Testing) Outline
• Demographics
• Level of knowledge research software developers have on testing
• Current testing practices in research software community
• Difficulties to test research software
• Compatibility of Commercial/IT testing techniques
• Improvement of the testing process in research software
31
0 2 4 6 8 10 12
ResourcesMake simpler
AutomationContinuous integration
CultureImprove code quality
AcknowledgementInfrastructure
More testsTraining
Testing Improvements
32Count
Discussion
• Researchers pose a clear goal of testing their project
• Complexity associated with the process needs further attention
• Make a culture of testing in the research software community.
• Providing proper training and resources can improve the testing process in research software.
33
Part #2
Peer Code Review in Research Software
34
Interview & Survey
35
Part #2 (Code Review) Outline
36
• Demographics
• Current code review practices in research software
• Impacts of the code review process in research software
• Difficulties develpers face during code review
• Potential areas of improvement in the review process
Part #2 (Code Review) Outline
37
• Demographics
• Current code review practices in research software
• Impacts of the code review process in research software
• Difficulties develpers face during code review
• Potential areas of improvement in the review process
0
5
10
15
20
25
30
Less than 1 year 1 year to lessthan 5 years
5 years to 10years
More than 10years
Years Worked
38
Coun
t
01020304050607080
Add newcode
Fix bugs Reportbugs
Maintainproject
Makestrategicdecisions
Other
Role
39
Coun
t
05
1015202530354045
1 -Reviewee
only
2 3 4 - Equally 5 6 7 -Reviewer
only
Balance as a Reviewee and Reviewer
40
Part #2 (Code Review) Outline
41
• Demographics
• Current code review practices in research software
• Impacts of the code review process in research software
• Difficulties develpers face during code review
• Potential areas of improvement in the review process
0
10
20
30
40
50
60
70
Less than10%
11% - 25% 26% - 50% 51% - 75% More than75%
Percentage of Code Undergo Review
42
Coun
t
05
101520253035404550
Less than 1 1 to 5 6 to 10 10 to 15 More than 20
Time Spent on Code review
43
Coun
t
Hour
0
5
10
15
20
25
30
35
Less than 1hour
Less than 1day
1 - 3 days 4 - 7 days More than 7days
Time For a First Response
44
Coun
t
05
10152025303540
Less than 1 day Less than 1week
Less than 1month
More than 1month
Time For a Final Decision
45
Coun
t
0 20 40 60 80 100 120
Other
Design
Code mistakes
Software quality
Problems Identified
46Count
0 10 20 30 40 50
Other
Problems identified
Positive feeling
Good feedback
Improved code quality
Knowledge sharing
Positive Experience
47Count
0 5 10 15 20 25
OtherUnresponsive author
Difficult taskHard to find reviewers
BottleneckDisagreements
Misunderstand criticismTakes too long
Negative Experience
48Count
Part #2 (Code Review) Outline
49
• Demographics
• Current code review practices in research software
• Impacts of the code review process in research software
• Difficulties develpers face during code review
• Potential areas of improvement in the review process
0 10 20 30 40 50 60
Other
Sustainability
Improve maintainability
Knowledge sharing
Improve code quality
Why Code Review is Important
50Count
0 5 10 15 20 25
OtherDocumentation
Better styleImproves reliabilityKnowledge sharing
Improve designBetter maintainability
More EyesImprove readability
Correctness
How Code Review Improves Code
51Count
05
10152025303540
Stronglydisagree
Somewhatdisagree
Neitheragree nordisagree
Somewhatagree
Stronglyagree
Decrease Code Complexity
52
Coun
t
Part #2 (Code Review) Outline
53
• Demographics
• Current code review practices in research software
• Impacts of the code review process in research software
• Difficulties develpers face during code review
• Potential areas of improvement in the review process
0 5 10 15 20 25 30 35
Other
Identify problems
Administrative issues
Understanding system
Understanding code
Challenges
54Count
0 10 20 30 40 50 60 70
Other
Takes too long
Developer egos
Participation
Finding right people
Phrasing comments
Time
Barriers
55Count
Part #2 (Code Review) Outline
56
• Demographics
• Current code review practices in research software
• Impacts of the code review process in research software
• Difficulties developers face during code review
• Potential areas of improvement in the review process
0 5 10 15 20 25
OtherFaster response
More timeMore training
IncentiveMore people
ToolingFormalizing process
Improvements
57Count
Discussion• Research software developers employ an informal code
review process
• Code review has an overall positive impact
• Most common difficulty reported by participants is finding time to to do it and understand other people’s code.
• Formalizing the review process by including more people, more training, and providing compensation could potentially improve the code review process.
58
Recommendations - Testing
59
• Provide enough training on software testing to all kinds of research software developers ranging from graduate students to experienced researchers
• Incorporate more tests that can solve specific needs of the research software
• Provide infrastructure support, for example, a public service for testing including many-tier pricing structure for machine time and a sophisticated testing dashboard
60
Recommendations - Testing
• Provide automation for setting tests and analysis of the results
• Improve continuous integration system to facilitate a better way of testing, especially, the incoming tests during down time
• Make a culture of testing in the team and encourage others by sharing the benefits from the experience of testing
• Improve the quality of the code so that developers can write tests easily
61
Recommendations - Testing
Recommendations - Testing
• Provide proper acknowledgement of developers for contributions in testing
• Make the testing process simpler so that it is easy to adopt in the project
• Provide enough resources to developers so that they can utilize the resources to develop test suits
62
Recommendations – Code Review
63
Recommendations – Code Review
• Make code review process more formal with a structured guideline for each step of the process
• Try to ensure at least one science review and one technical review
• Include automatic tools in the code review process and train your peer reviewers the best practices to use the tool
64
• Encourae more people to participate in the review process and allocate some time to do the review
• Provide incentives or rewards to reviewers to participate in code review
• Allocate sufficient time in the development process to perform code review
• Provide faster feedback to any incoming review request
65
Recommendations – Code Review
• Train reviewers on how to phrase good feedback
• Train developers to forget their egos and accept comments from the reviewers to improve their code
• Make the overall code review process faster
• Provide necessary support from the administrative level that encourages people to participate in the code review process
66
Recommendations – Code Review
AcknowledgementDr. Jeffrey CarverUniversity of Alabama
Dr. George ThiruvathukalLoyola University Chicago
Dr. David BernholdtOak Ridge National Laboratory
Dr. Hai Ah NamLos Alamos National Laboratory
Dr. Danny PerezLos Alamos National Laboratory
Dr. J. Dave MoultonLos Alamos National Laboratory
Dr. Roland HaasNational Center for Supercomputing Applications
Dr. Gabrielle AllenNational Center for Supercomputing Applications
Dr. Daniel KatzNational Center for Supercomputing Application
NSF grant 1445344
67
68
Nasir [email protected]