international conference on software and system...
TRANSCRIPT
Impact of context switching and work interruptions on software development processes
International Conference on Software and System Processes
July 5-7 2017, Pierre and Marie Curie University, Paris, France
Alexey Tregubov*, Barry Boehm, Jo Ann Lane, Natalia Rodchenko
University of Southern California
* presenting author
Motivation Cross-project multitasking overhead is often observed in
• Matrix organizations - resources are shared between several projects for
better resource utilization.
• Multiple releases of a product – if a product is released more than one
time, resources are shared between maintenance of previous releases
and a new version (typical for small mid-size teams).
• System of System (SoS) environments – in SoSs, if a constituent system
is developed for several customers (e.g. different software
distributions/releases for each customer), resources are shared between
different work contexts.
2
Motivation Cross-project multitasking overhead is often not accounted when
projects are estimated. This may cause underestimation in project
planning.
3
Multitasking in work environment
4
Work productivity
Corporateculture
OfficeEnvironment
Personality
Personal process
Cross-project multitasking
Management Psychology
Work scheduling, technical processes
imp
acts
Work quality
Work interruptions
Work interruptions
5
Cross-project multitasking in software development
When developers work on several projects at the same time
(over a week or even a day), they switch between them
spending some time on context switching.
Not all context switching is bad, but we only focused on
excessive switching between different projects, which is
interrupting work and causing productivity decline.
Context switching between projects cost/time consists of
• physical time to switch– switch between repositories,
DBs, servers, etc. takes time
• cognitive context switching –getting into ‘flow mode’
takes time
6
Research questions and hypothesis
RQ1. What is the quantitative effect of cross-project multitasking
overhead on development effort and quality?
• H1.a The number of cross-project interruptions is (not) linearly proportional
to the number of projects.
• H1.b The G. Weinberg’s heuristic is (not) applicable for cross-project
multitasking overhead estimation in software development teams.
• H1.c The number of cross-project interruptions is (not) linearly proportional
to the number of reopened tasks (rework).
RQ2. How can COCOMO II model be improved to account for cross-
project multitasking overhead?
• H2. Using the multitasking effort multiplier (MEM) in the locally calibrated
COCOMO II model can improve prediction accuracies of the COCOMO II.
7
Gerald M. Weinberg’s heuristic
8
0
10
20
30
40
50
60
70
80
1 2 3 4 5
% o
f ef
fort
sp
en
t o
n m
ult
itas
kin
g
Number of projects at the same time
Weinberg's heuristic
Research methodology
9
COCOMO IIcalibration
Compute MEM for each
data point
Test hypothesisH2a,b
Source data: work logs of different project groups
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
Collected data (work logs and schedules)
Ind
ust
ry p
roje
cts
Stu
den
ts’ p
roje
cts
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
R1R2R3R3
Project 1 schedule
Extended data set from simulation
Sim
ula
tio
n
Test hypothesis H1a,b,c
Count cross-project interruptions for each data point
Compute MEM for each
data point
Count cross-project interruptions for each data point
Compute quality metric
Evaluate model prediction accuracies
RQ1: impact of multitasking on effort and quality
RQ2: COCOMO II calibrationlink between data source and data
transformation/manipulation
Data collection
Industry projects:
• Two groups of projects, one year each
• Work logs in JIRA – hours reported daily for each task in each project
• Schedules were updated every 2-3 days
• SLOC added and changed for each project
577 students projects
• 10 teams/projects, 5-11weeks development phase
• Work logs in JIRA – hours reported weekly for each task in each project
• Schedules were updated every 2 weeks
• SLOC added and changed for each project
10
Cross-project work interruptions (CSCI577 class projects)
11
R2 0.61110
S 2.418
p 0.0003
n 29
Cross-project work interruptions (industry)
12
R2 0.4563
S 2.198
p <0.005
n 154
Cross-project work interruptions (CSCI577 class projects)
13
R2 0.785
S 0.617
p 0.0001
n 21
Cross-project work interruptions (industry)
14
R2 0.48
S 379
p <0.005
n 142
Cross-project work interruptions (CSCI577 class projects)
15
Cross-project work interruptions (industry)
16
Multitasking effort multiplier (MEM)
𝑬𝒇𝒇𝒐𝒓𝒕 = 𝑨 × [𝑺𝒊𝒛𝒆]𝑩+ 𝑺𝑭𝒋𝟓𝒋=𝟏 × 𝑬𝑴𝒋𝝎
𝟏𝟕
𝒋=𝟏
×𝑴𝑬𝑴
𝑀𝐸𝑀 = 𝑓 𝐼, 𝑅, 𝐸 =𝐸
𝐸 − 𝐼𝑅=1
1 − 𝑅𝐼𝐸
=1
1 − 𝑅𝐼/𝑡𝐸/𝑡
Cross-project multitasking effort multiplier (CSCI577 class projects)
18
R2 0.11
S 0.2
p 0.008
n 29
Cross-project multitasking effort multiplier (industry)
19
R2 0.45
S 0.01
p <0.005
n 154
Comparison with Weinberg’s heuristic (CSCI577 class projects)
20
Comparison with Weinberg’s heuristic (industry)
21
Impact on quality (CSCI577 class projects)
22
R2 0.44
S 2.25
p 0.051
n 9
Impact on quality (industry)
23
R2 0.71
S 1.01
p 0.004
n 9
Results: RQ1 H1.a. The number of cross-project interruptions is (not) linearly proportional to number
of projects.
• Tested negative
• For a given group of industry projects and CSCI577 projects the number of cross-project interruptions is not linearly proportional to number of projects. The linear correlation is relatively weak (R^2 is between 0.46 and 0.61 in all tests).
H1.b. The G. Weinberg’s heuristic is (not) applicable for cross-project multitasking overhead estimation in software development teams.
• Tested negative
• For a given group of industry projects and CSCI577 projects cross-project multitasking overhead cannot be evaluated with the G.Weinberg’s heuristic because it overestimates effort for this type of work interruptions.
H1.c. The number of cross-project interruptions is (not) linearly proportional to the number of reopened tasks.
• Tested positive
• For a given group of industry projects cross-project interruptions are linearly correlated with the number of reopened JIRA tickets.
H2. The multitasking effort multiplier (MEM) can be automatically evaluated based on work log observations.
• Tested positive
• Work log analysis algorithm was implemented as part of the DES simulation framework and used to compute results presented above.
24
RQ2: Calibrated COCOMO II prediction accuracies
25
COCOMO II Local
calibration of
COCOMO II
Local calibration of
COCOMO II with MEM
PRED(.20) 32% 56% 63%
PRED(.25) 37% 58% 75%
PRED(.30) 38% 61% 81%
Results: RQ1
26
Amount of work Interruptions
Number of projects
Multitasking Overhead (Metric: MEM)
Quality(metric: reopened tasks
or grade deduction)
Effort
Reimmersion time
Eval
uat
ion
Evaluation
H1.a weakcorrelation
H1.c positive correlation
H1.b no correlation
Causal Graphical Model – results summary
Implied causality or functional relationship
Hypothetical association or causality
Observation
Observation & Evaluation
Observation Observation
ObservationEvaluation
Individual level Individual and team level
Individual and team level
Team level
Individual level
Individual level
Number of projects
Metric or measurment
H1.a – tested by hypothesis
Challenges and threats to validity
Reimmersion time does not necessarily capture the time needed to restore
work context.
• Used the average minimum estimation of the reimmersion time. In the worst case, the values
found in this study serve as an lower bound for multitasking overhead.
Self-tracking of the reimmersion time and hours devoted to projects may not be
accurate
Students may inaccurately report effort in Jira
• To address this threat, students effort reports in Jira were graded every week. Additionally,
weekly reminders were sent out about the reports.
Academic environment, where graduate students of CSCI577 class worked, is
different from developers in industry.
• To limit the impact of deadlines in other classes, students in the selected sample were
enrolled in the same classes (e.g. students who took 3 classes were all enrolled in software
engineering class, algorithms class and AI class).
27
Future work
The approach for counting work interruptions and evaluating their
impact on effort can be applied to other types multitasking. To do so
we need to develop methods for measuring the number of work
interruptions caused by certain type of multitasking.
The further accuracy evaluation of models with MEM (any
COCOMO family model with MEM) should be done on a bigger
sample of projects.
The work log analysis tools, we used in the research, can be
integrated with project tracking systems such as Atlassian JIRA to
provide real-time information about work interruptions and their
impact on productivity (e.g. calculate MEM).
Further research should be done on how to schedule and
organize work to reduce negative effects of work interruption
on productivity
28
Q&A, discussion
29
References
1. Meyer, D. E. & Kieras, D. E., 1997. A computational theory of executive cognitive processes and multiple-task performance: Part 2. Accounts of psychological refractory-period phenomena. Psychological Review, 104, 749-791.
2. Weinberg, G.M., 1993. Quality software management. New York.
3. DeMarco, T. and Lister, T., 2013. Peopleware: productive projects and teams. Addison-Wesley.
4. American Psychological Association, 2006. Multitasking: switching costs. URL: http://www. apa. org/research/action/multitask. aspx.
5. Dzubak, C.M., 2008. Multitasking: The good, the bad, and the unknown. The Journal of the Association for the Tutoring Profession, 1(2), pp.1-12.
6. Delbridge, K.A., 2000. Individual differences in multi-tasking ability: Exploring a nomological network. Unpublished Doctoral Dissertation, University of Michigan.
7. Tregubov, A. and Lane, J.A., 2015. Simulation of Kanban-based Scheduling for Systems of Systems: Initial Results. Procedia Computer Science, 44, pp.224-233.
8. Leonard-Barton, D. and Swap, W.C., 1999. When sparks fly: Igniting creativity in groups. Harvard Business Press.
9. Jett, Q.R. and George, J.M., 2003. Work interrupted: A closer look at the role of interruptions in organizational life. Academy of Management Review, 28(3), pp.494-507.
10. Tregubov, A. and Lane, J.A., 2015. Simulation of Kanban-based Scheduling for Systems of Systems: Initial Results. Procedia Computer Science, 44, pp.224-233.
30
References
11. Dingsøyr, T., Nerur, S., Balijepally, V. and Moe, N.B., 2012. A decade of agile methodologies: Towards explaining agile software development. Journal of Systems and Software, 85(6), pp.1213-1221.
12. Boehm, B. and Turner, R., 2003. Balancing Agility and Discipline: A Guide for the Perplexed, Portable Documents. Addison-Wesley Professional.
13. Larman, C. and Vodde, B., 2008. Scaling lean & agile development: thinking and organizational tools for large-scale Scrum. Pearson Education.
14. Poppendieck, M. and Poppendieck, T., 2007. Implementing lean software development: from concept to cash. Pearson Education.
15. NDIA-National Defense Industrial Association, 2010. Top Systems Engineering Issues In US Defense Industry. Systems Engineering Division Task Group Report, http://www. ndia. org/Divisions/Divisions/SystemsEngineering/Documents/Studies/Top% 20SE% 20Issues, 202010.
16. Turner, R., Shull, F., Boehm, B., Carrigy, A., Clarke, L., Componation, P., Dagli, C., Lane, J.A., Layman, L., Miller, A. and O'Brien, S., 2009. Evaluation of Systems Engineering Methods, Processes and Tools on Department of Defense and Intelligence Community Programs-Phase 2 (No. SERC-2009-TR-002). SYSTEMS ENGINEERING RESEARCH CENTER HOBOKEN NJ.
17. http://www.businessdictionary.com/definition/matrix-organization.html
18. https://hbr.org/1978/05/problems-of-matrix-organizations
19. http://www.joelonsoftware.com/articles/fog0000000022.html
20. Mark, G., Gonzalez, V.M. and Harris, J., 2005, April. No task left behind?: examining the nature of fragmented work. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 321-330). ACM.
31
References
Images courtesy: http://kursevi.edukacija.rs/wp-content/uploads/2015/07/multitasking-efikasnost.jpg
http://mercercognitivepsychology.pbworks.com/f/1385057522/Multitasking.jpg
http://www.projectengineer.net/wp-content/uploads/2012/12/life_cycle.png
32