Broadcast vs. Unicast Review Technology:Does it Matter?

Armstrong Foundjem, Foutse Khomh, Bram AdamsSWAT–MCIS, École Polytechnique de Montréal, Québec, Canada

{a.foundjem, foutse.khomh, bram.adams}@polymtl.ca

Abstract—Code review is the process of having other teammembers examine changes to a software system in order toevaluate their technical content and quality. Over the years,multiple tools have been proposed to help software developersconduct and manage code reviews. Some software organizationshave been migrating from broadcast review technology to a moreadvanced unicast review approach such as Jira, but it is unclearif these unicast review technology leads to better code reviews.

This paper empirically studies review data of five Apacheprojects that switched from broadcast based code review tounicast based, to understand the impact of review technologyon review e�ectiveness and quality.

Results suggest that broadcast based review is twice fasterthan review done with unicast based review technology. However,unicast�s review quality seems to be better than that of thebroadcast based. Our findings suggest that the medium (i.e.,broadcast or unicast) technology used for code reviews can relateto the e�ectiveness and quality of reviews activities.

Index Terms—Code Review, Patches, Apache, Empirical Study,Medium, Bug-inducing Changes , Quality Assurance

I. I�����������Peer Code Review (PCR) also known as Code Review

(Review) is an integral part of the quality assurance process ofsoftware projects [1] [2] [3]. The main objective of Review isto systematically examine source code changes (SCC) writtenby other programmers in order to verify their correctness andquality before integration into the Version Control System(VCS) (e.g., Git).

While the formal code inspection activities proposed byFagan [4] in 1976 were heavyweight, nowadays the softwareindustry has adopted lightweight variants of Review, supportedby modern tools and technologies [5] [6] [7]. In order to im-prove quality assurance [2], recent research has even proposedautomatic Review tools [8] to reduce human e�orts [9].

Currently, there are two main categories of technologies forReviews. The first category uses a broadcasting technologyapproach of communication whereas the second category usesa unicast technology approach of communication.

• In the broadcast technology approach, when a patch issubmitted for review, all those who are subscribed to themedium where the patch is submitted can see the patchsee Figure 4, and anyone can review the patch. Further-more, any subsequent discussion concerning the patchis visible to all the members who are subscribed to themedium; hence broadcasting. An example of a broadcasttechnology is the developer’s mailing-list [10] [11].

• In the unicast technology approach, when a patch issubmitted for review, it is visible only to a targeted (a setof dedicated) group of individuals who are subscribedto the communication medium and who have beenassigned to the patch for review see Figure 5. Only thosetargeted individuals can review the patch and followupdiscussions concerning the patch. An example of theunicast approach is an issues tracker [12], such as Gerrit,or JIRA etc. In this paper we focus on JIRA.

Many software development organizations such as the ApacheSoftware Foundation (ASF) are migrating from the broadcasttechnology approach review environments [13] [14] [15] to theunicast technology approach [5] [6] [16] [17] issues trackerenvironments such as Gerrit, Jira and the like. However,despite this keen interest in the unicast technology Reviewplatforms, there is no evidence that they make code reviewactivities better. Although there have been many studies oncode review [5] [18] this past decade, to the best of ourknowledge, none of these studies have performed a head-to-head comparison based on the technology used for reviewingcode, in terms of e�ectiveness, e�ort, and e�ciency of Reviewactivities.

This paper aims to fill this gap. More specifically, we aim toexamine if the technology (i.e., broadcast vs. unicast) throughwhich patch reviews are performed relates to e�ort, e�ective-ness and e�ciency of review activities. In order to achievethis goal, we use data from five open source projects from theApache software foundation (i.e., Flex, HBase, Hive, PIG, andSVN), that have switched from broadcast technology approachto unicast technology approach (issues tracker), which in thiscase is JIRA. We address the following research questions(RQs):

RQ1 - Is review e�ort related to the review medium used?Result: We measure the review e�ort for a patch usinginformation about the number of developers involved in thereview, the number of rounds necessary to review the patch,and the number of review requests for the patch. Our resultsshow that in general, patches reviewed on unicast technologyundergo more revisions than patches reviewed on broadcasts.

RQ2 - Is the e�ectiveness of a patch reviewing processrelated to the medium used?Result: To capture the e�ectiveness of a review process, wecompute the proportion of reviewed patches that experienced apost release bug, using the SZZ algorithm [19]. Results show

that patches reviewed through broadcast technology are morelikely to experience a post release bug than those reviewedon unicast technology. We examined the e�ect of size on thisresult and found no significant di�erence between the sizes ofpatches reviewed respectively through broadcast and unicasttechnology.

RQ3 - Is the e�ciency of a patch review process related tothe medium used?Result: We measure the e�ciency of a patch review processusing two metrics: the response delay in days after a patchreview request is submitted and the period of time between thesubmission of a patch review request and the integration of thepatch into the VCS. Results show statistically significant di�er-ences between the response delay (RD) and the review length(RL) respectively, of patches reviewed on broadcast technologyand those reviewed on unicast technology. However, we foundno correlation between the size of patches against the responsedelay and the size of patches against the length of the reviewperiod (RL).

The rest of this paper is organized as follows: Section Vreviews the related literature, Section II describes the approachused to collect, filter, compute and analyze data. In section III,we present and discuss the results of our research questions.We discuss the threats to validity of our study in section VI.Finally, we conclude the paper in section VII.

II. A�������In this section, we present the steps of our approach.

First, we select stable Apache projects that were using thebroadcast technology approach for reviewing patches and atsome point in time, switched to a unicast technology approachfor reviewing patches. Then, we collect and analyze the datarequired to answer our research questions. We also interviewedreviewers of the Apache Software Foundation (ASF), usinga questionnaire [20] that we designed. Figure 1 shows anoverview of our approach and Figure 2 illustrates the time pe-riods that are considered in this study. The middle gray regionindicates the window (threshold) period where the switchedfrom broadcast technology approach to unicast technologyapproach occurred.

A. Projects SelectionThe Apache ecosystem has projects that were using the

broadcast technology approach for reviewing patches and lateron switched to unicast technology approach, moreover, Rigbyet al. [21] [22] have studied the Apache’s broadcast basedreviews, and Bettenburg et al. [23] have also studied projectsin the Apache Ecosystem. Therefore, the Apache ecosystemappeared to be the ideal choice for our study. To identify theperiod during which the projects transitioned from broadcasttechnology approach to unicast technology approach, we ex-amined the volume of patches reviewed on both platforms(broadcast and unicast technologies) over a given period oftime and observed that in the gray region shown in Figure 2,there were a drastic decrease in the number of patches thatwere processed using the broadcast technology approach and

Figure 1: Our Approach.

Figure 2: Our Approach: time-line.

a sharp increase of the number of patches processed usingthe unicast technology approach. The intersection point wherebroadcasts crosses over unicast forms the transition point forthe projects; an example is shown by the gray region inFigure 2 and Figure 3 for the PIG project.

To identify the transition period of the PIG project forexample, we computed the proportion of patches reviewed onbroadcast platforms and the proportion of patches reviewed onunicast platforms between January 2012 to December 2012.We observed that initially, reviewers reviewed between 33 to42 patches through the broadcasts each week, in the 25th weekwe observed a sudden decrease of patches, that is, below10 patches per week over a period of 6 months (July toDec), see Figure 3. At the same time unicast’s reviewers wereexperiencing a drastic increased of patches reviewed fromabout 10 to 35 patches weekly, see Figure 3.

Based on our observation, we defined a window period (therectangle in Figure 3), which also represents the rejectionwindow, i.e., we reject all the patches within this periodfor this study. In the case of the PIG project for example,we consider broadcast patches up-to the left boundary ofthe window (25th week) and unicast patches from the rightboundary (30th week) onward to the limiting period and do thesame for other projects in this study. Within this window, thatis between the (25th and 30thweek), since there is an overlapbetween patches using broadcast and patches using unicast,we rejected all the patches within this window’s region soas to have a clean dataset. This window takes into accountthe period immediately before and immediately after switchingfrom broadcast to unicast.

To determine the window, considering the PIG project forexample, we computed the intersection point of the two curves(i.e., the proportion of patches reviewed on broadcasts and theproportion of patches reviewed on unicast), and uses it as areference point to know exactly when the project switchedfrom broadcast to unicast.

This reference point is based on the volume of previouslysubmitted patches that were reviewed using broadcasts (periodbefore switch) and patches reviewed using unicast (periodafter the switched). For the PIG project, the reference pointis on the 27th and a half weeks, which corresponds to thetransition period between July and August. As can be seenfrom Figure 3. Then, we observe the period when the patchesdrastically decrease when using broadcast; the left boundaryof the window (25th week), and the period when patchesdrastically increase when using unicast; the right boundaryof the window (30th week), this gives us the window size ofa project (PIG project), with a cuto� of less than 10 patchesper week (the intersection point where broadcast crosses overunicast). Five projects of the Apache ecosystem had referencepoints where reviewers did less than 10 patches per weekwithin this window that we defined. Therefore, we selectedthese projects for our study: HBase, Hive, PIG, SVN and Flex.

Figure 3: PIG Transition from broadcast to unicast

B. Broadcast Technology vs. Unicast Technology ApproachAfter selecting the projects needed for this study, our next

step was to understand the structures of both the broadcast andunicast approaches of reviewing patches.

1) Broadcast Technology Approach: broadcasts archivesare arranged as .mbox files [24], categorized per year, thread,date and author, with the associated number of threads. Thisstructure is the same for the broadcast archives of all the fiveprojects. Also, the patches are attached to the emails. We usea python script to download the .mbox files for all the fiveprojects within the given period of our study as shown inTable III.

2) Unicast Technology Approach: Issue tracker [25] arealso areas where source code changes are reviewed before

committing to the code-base repository. Moreover, they areenablers for the review process in the sense that, they capturecomments regarding the proposed changes of the source codeand enable ongoing discussions on the topic. Since all com-mits have a specific identification number (ID), issue trackersystems use this ID to link revisions to commits, in addition,they keep track of each revision, the author, date, and historyof the code changes.Most issue tracker environments, like unicast, provide webportal to visualize the reviewing process and by default,groups every revision on the same page. Figure 5 (courtesyof Atlassian 1) shows the work flow and status of the codereview process using di�erent colors to indicate the status.

Figure 4: Work-flow and Status in broadcast environment.

Figure 5: Work-flow and Status in unicast environment.

C. Data ExtractionAfter identifying the data sources for both broadcasts and

unicast, we used the following tools to extract the data asneeded and created a database to store the data.

Broadcast Technology: We used MailboxMiner2 [26] (aJava application) to create a relational postgreSQL databasewith tables containing the email messages and their metadata.We analyze email attachments, to extract information aboutthe patches.

Emails are grouped per thread, where each email is a replyto an earlier email. Each thread contains one or more emailswith di�erent revisions (versions) of a code patch, hence, weneed to distinguish the emails with patch revisions from thosewith review comments (the other emails).

Basically, since we need all revisions of a patch, we usedthe identification number of patches to group all the thread of

1https://www.atlassian.com/software/jira

the same patch. For example, this attachment from an email:789.patch has a key word in the message body HBASE-789,we used this keyword to link patches to Git [27]. In the end, foreach patch sent to a broadcast, we can reconstruct all revisionsof the patch as well as their general comments. Of course,since emails can contain any kind of text, we need to eliminatenoise, such as a reply that does not contribute anything to theReview. However, emails automatically generated by unicasthave a consistent pattern attached to them, which allows toidentify a particular commit. For example, HBase-960, meansa commit from the HBASE project number 960. Same forPIG-XXX, Flex-XXX, and SVN-XXX, and if there is a replyfrom a reviewer, the message will contain “RE:". With thehelp of this specific Identification, we were able to link emailcontributions to the Git repositories [23] with the help of thismirror site [27]. We used git commands such as git di�, gitblame, and git log to extract various information about thepatches reviewing process.

Unicast Technology: We could extract and export data fromunicast in Excel format. We wrote a Python script to processthe data.

D. Structure of the Questionnaire

We interviewed reviewers from the Apache Software Foun-dation using a questionnaire2, containing both open-ended(aims at asking the reviewers to formulate their own answer)and closed-ended (aims at asking the reviewers to chose ananswer from a set of given options) questions. We arranged thequestions logically; connecting questions from one to the next,from a general to a more specific perspective. In some of thequestions, we wanted to know what motivated the switch frombroadcast to unicast review technology, and the di�culties thatthe reviewer faced in changing their review process. Moreover,we go further to know details about the review process, basedon their individual experiences.

After designing the questions, we sent out the questionnaireform [20] to selected Apache reviewers. The procedure toselect the Apache reviewers to participate in this survey isas follow. First, for the five projects we used in this study, weselected the top 10 active reviewers per project (participantsmust have contributed in reviewing patches). These 10 activereviewers per project gave us 50 reviewers in total. Then wefiltered 50 reviewers by selecting those who used the broadcastto review patches and then continue reviewing using unicastover the study period. One of the project (SVN) had seven (7)of such reviewers in total, so we decided to normalized all theprojects to seven top reviewers per project. That is how weended up with 35 reviewers (7-reviewers x 5-projects) in totalfor our questionnaire.

This selection ensures us that the surveyed reviewers cangive us meaningful feedback, which would represent the entireperiod of our study.

2https://goo.gl/forms/AgQMasO7mne6G2jg1

Table I: Metrics used to captured review e�ort, e�ectiveness,and e�ciency.

Type Metric Description

E�or

t Number of Reviewers(NR)

Number of developers who Par-ticipated in the review of apatch

Number of revisions(NV)

Number of rounds necessary toreview a patch [1]

Review queue (RQu) Number of pending reviews re-quest per reviewer; reviewerworkload [28]

E�ec

tiven

ess Post review bugs Proportion of reviewed patch

that experienced a post releasebug. To identify a bug intro-duced after a patch review, weuse the SZZ algorithm [19]

Median review rate(MRR)

This is the number of lines ofcodes (KLOC/week) for eachreview (also known as the Av-erage review rate) [1] [30] [31].Since we can’t claim thatthe review rates are symmet-rically distributed, we will usethe median review rate in-stead. The median review ratemetric aims to examine ifreviewers are maintaining agood code-reviewing cadence,for example, not going above400KLOC/hr or not performingbelow 200LOC/hr [30] [32].

E�ci

ency Review length (RL) Time (days) from submission

of a review request to the in-tegration of the patch [1]

Response Delay (RD) Number of delay in days aftera patch review request is sub-mitted [1]

Size Size of a patch [28]

E. Metrics ExtractionBased on our RQs and existing work [1] [28] [29], we

extracted the metrics presented in Table I. The metrics aregrouped according to the RQs.

RQ1 - We capture the e�ort required to review patches, seeTable I.

RQ2 - In this RQ, we defined e�ectiveness as the ability ofreviewers to detect bug-inducing commits during code reviews.We use the SZZ-algorithm [19] [33] [34], that automaticallyidentifies fix-inducing commits by linking bugIDs found incommit messages with bugs repository using keywords such as“FIXED" and using regular expressions. This algorithm backtracks lines changed through the revision history to the pointof its most recent change. After locating the keywords, we usethe di� tool to know what changed in bug-fix, and outputsa hunk. In addition to tracking post review bugs, we alsocomputed the median review rate for each Review platform(in KLOC/week) [31].

RQ3 - We capture the e�ciency of a review process usingthe review length and the response delay metrics describedin Table I. To compute the response delay, we calculate thedi�erence from the reply date of the first revision. To capture

the review length, we compute the time period between thesubmission of a review request and the integration of thereviewed patch into the version control system.

F. Data AnalysisTo address our research questions (RQs), we test the fol-

lowing null hypothesis for each metric m:H0 : There is no significant di�erence between the valueof metric m for patches reviewed on broadcasts and thosereviewed on unicast.We use the Mann-Whitney U test to test H0. The Mann-Whitney U test is a nonparametric statistical test used forassessing whether two independent distributions have equallylarge values. Nonparametric statistical methods make no as-sumptions about the distributions of the assessed variables.We set ↵ to 0.05. Moreover, we consider a Mann-Whitneytest result to be statistically significant if and only if the p-value is below ↵. Additionally, we compute the e�ect-sizeof the di�erence using Cli�’s � (cohen 1998) [35], which isalso a non-parametric statistic test, e�ect size measure, whichmeasures how often values in one distribution are larger thanvalues in another distribution. Cli�’s � d values lies between[-1,1] and is considered negligible (N) for |d|

Table II: p-value and Cli�’s-� results for our studied metrics referred in table I.

Projects NR NV RQu MRR RL RD Size SZZp-value Cli�’s � p-value Cli�’s � p-value Cli�’s � p-value Cli�’s � p-value Cli�’s � p-value Cli�’s � p-value Cli�’s� p-value Cli�’s�

Flex 0.843 N 0.016 L 0.012 L 0.021 L 0.015 L 0.019 L 0.924 N 0.015 MHBase 0.891 N 0.021 L 0.017 L 0.017 L 0.013 L 0.016 L 0.831 N 0.039 MHive 0.902 N 0.011 L 0.031 L 0.014 L 0.021 L 0.011 L 0.913 N 0.031 MPIG 0.931 N 0.017 L 0.021 L 0.011 L 0.015 L 0.012 L 0.917 N 0.017 MSVN 0.915 N 0.013 L 0.019 L 0.031 L 0.017 L 0.022 L 0.821 N 0.008 M

0.00

0.25

0.50

0.75

1.00

0 5 10 15# Reviews

% R

evie

wer

s Tech

JIRA

ML

(a) PIG Project

0.00

0.25

0.50

0.75

1.00

0 5 10# Reviews

% R

evie

wer

s Tech

JIRA

ML

(b) Flex Project

0.00

0.25

0.50

0.75

1.00

0 5 10# Reviews

% R

evie

wer

s Tech

JIRA

ML

(c) Hive Project

0.00

0.25

0.50

0.75

1.00

0 5 10# Reviews

% R

evie

wer

s Tech

JIRA

ML

(d) Hbase Project

0.00

0.25

0.50

0.75

1.00

0 5 10# Reviews

% R

evie

wer

s Tech

JIRA

ML

(e) SVN Project

Figure 6: Review survival curve for HBase and SVN projects.

Table III: Projects transitional periods and reviews.

Projects Study-Period Transition Reviewsunicast - broadcasts

Flex 2011-2015 11-2013 193 - 195HBase 2011-2015 08-2013 212 - 222Hive 2011-2015 05-2013 216 - 177PIG 2011-2015 07-2013 280 - 291SVN 2010-2014 10-2012 183 - 179

Table IV: X percent of reviewers did at least Y reviews.

Medium pct Projects (number of reviews)

Flex HBase Hive PIG SVN

broadcasts 100% 4 4 4 5 550% 7 8 7 10 820% 9 9 9 12 9

unicast 100% 5 5 5 7 950% 8 9 9 12 1220% 9 11 11 20 13

Findings: The proportion of reviewed patches that experienceda future bug is lower for unicast than broadcast.

To ensure a fair comparison between patches reviewed onboth technologies; broadcasts and unicast, we filtered the dataas mentioned in RQ1 to have the same time window before

and after, and observe post review bugs for each patch. Afterthat, we varied the size of this time window and computed oneach window the accepted patches that experienced a futurebug. Unicast technology Reviews are more e�ective in termsof identifying buggy patches than reviews done on broadcasttechnology. Figure 7 presents the distributions that we ob-tained. The proportion of reviewed patches that experienceda future bug is lower for unicast. We also report results of theMRR in Table II, which is higher for broadcast than unicast.Overall, there is a statistically significant di�erence betweenthe MRR on broadcasts and unicasts, hence, we reject our nullhypothesis for this metric indicating that unicast is better thanbroadcast in terms of e�ectiveness.

Reviews performed on unicast technology are more ef-fective in terms of catching bugs than those performedon broadcast technology.

RQ3 - Is the e�ciency of a patch review process related tothe medium used?Motivation: Early studies showed that patch review is timeconsuming and slow [38]. Studies also show that participationin code review influences quality [28]. This research questionanalyzes whether the medium on which reviews are conductedrelates to the e�ciency of the patch review process.

Figure 7: Proportion of reviewed patches that experienced a future bug for di�erent time windows.

Approach: To answer this research question, we compute twometrics: the review length (days) (RL), and the response delay(RD), see Table I. To control for the e�ect of patch size on ourresults, we also computed and compared the size of patchesreviewed on broadcast and unicast technologies.

Findings: There are statistically significant di�erences be-tween the response delay and the review length respectively ofpatches reviewed on broadcast and those reviewed on unicast.Broadcast has a short response delay and a shorter reviewlength.

Results presented in Table II show that there is a statis-tically significant di�erence between the review length (RL)of patches on broadcast and unicast platforms, and also forthe response delay (RD). Moreover, the median value of thereview length is lower on broadcasts (i.e., 20 days, which isless than half the value of the median for unicast).

Moreover, there is no statistically significant di�erencebetween the size of patches reviewed on both broadcast andunicast technologies, therefore, size is not the factor behindthe observed di�erences in response delay and review length.

We compared the sizes of patches reviewed on the two plat-forms (i.e., broadcast and unicast) because the time requiredto review patches could depend on the size of patches underreview.

Our results also show that patches are reviewed on broadcastplatforms faster than on unicast platforms (in terms of reviewlength and response delay). This is irrespective of the size ofthe patches.

We found no di�erence between the size of patches re-viewed on broadcast and unicast platforms. We also foundno correlation between the size of patches and the responsedelay, or the size of patches and the review length. However,we found a statistically significant di�erence between theresponse delay (and the review length) of patches reviewedon broadcast platforms in comparison to those reviewed onunicast platforms.

Overall, we conclude that the medium on which codereview is performed can a�ect the e�ciency of thereview process.

A. Selected Results from Survey.To analyze the response from the Apache reviewers who

answered our questionnaires, we grouped the response (A) ofeach question (Q) in the questionnaire and then did a summaryper question. Then, we compared them with the results of ouranalysis.

Among the 35 copies of the questionnaires we sent, wereceived 20 complete responses, 5 incomplete responses, and10 non-responses over the period of 45 days. Moreover, theresponse proportion was equally distributed among broadcastand unicast approach. There were 9 questions in all, includingboth open-ended and closed-ended.

Let us highlight some key comments and observations fromsome selected Apache reviewers on their experts’ opinions.

• Q: What motivated the switch from broadcast to unicast?A: 12 of the reviewers said that the broadcast weregood for discussion (functional/design/customer relatedannouncements/release etc.). However, when it comesto reviewing code, tracking tasks/issues – such as whatrelease of the product they are targeted for, whether anissue has been fixed (and if so, which release or releases,etc.), whom an issue is assigned to, who reported theissue, etc.. – the unicast system is much convenient forreviewers than broadcast, since it reports all (meta)dataabout a particular patch on one web page.

Unicast makes it much easier for the reviewer toleave comments on specific blocks or lines ofcode, and to visualize the exact changes madeby a di� for example. Also, unicast has lesstra�c (volume of patches circulating) among thereviewers.

Figure 8: Size of patches.

• Q: How hard was it for reviewers to change their reviewprocess from broadcast to unicast?A: The reviewers unanimously agreed on the switched.Furthermore, they said it was pretty easy learning andusing unicast without any interruption to their task.

• Q: Based on their personal experiences with both re-viewing technologies, what was their main advantages ordisadvantages?A: The reviewers answered that, unicast technologymakes it easier to review patches, track progress onbugs/issues, look up details on old issues, easier to makerelease notes on what has been fixed, and easier toorganize releases.However, since it has these many feature, sometimes it cantake longer to review patches as the reviewer might wantto explore the many options while reviewing patches.Whereas with the broadcast, reviewing patches keeps youfocus on the patches and discussions around the patches.On the other hand, due to the broadcasting nature ofbroadcast technology, every reviewer see the patch, whichfacilitates the learning process for new developers whojoin a team (speeding them up quickly), as it animatesdiscussions around code structure, style, and architecture.

New developers learn about the code structurefaster with broadcast than using unicast. Thetra�c of patches circulating on broadcast is high,because it circulates among all those who aresubscribed to the broadcast medium.

IV. D���������

In this section we discuss our results in more details.We also comment on how our findings agree with Apachereviewers experience using both the broadcast and unicasttechnologies. First, we comment on the three categories, thatis: E�ort, E�ectiveness and E�ciency.

For RQ1, We address E�ort. We measure e�orts based onthe metrics mentioned in Table I for both the broadcasts andunicast. The statistical test results for the number of reviewers(NR) (see Table II) were a bit expected since we selectedprojects that were first using broadcasts and then switchedto unicast; it is no surprise that the number of reviewersremained almost the same after the transition from broadcastsand unicast. This is also an interesting point to note for thisstudy, we fixed this metric (NR) as a constant and study thee�ect of the number of revisions (NV) and the review queue(RQu). These two metrics show significant di�erences betweenthe broadcast and unicast technologies. We did not consideredthe learning curve for developers, that is the time to learnthe new tool during the switch from broadcasts to unicast.We also didn’t considered the cultural change. Nevertheless,we analyzed several comments but found none that expressesany such di�culties or challenges. Therefore, we conclude thatbased on e�ort, unicast seems to be a better tool for Reviewthan broadcasts.

For RQ2, we address E�ectiveness. From the results ofthe proportion of reviewed patches, which experienced a futurebug and the median review rate (see Table II and Figure 7),we observe that patches reviewed through broadcasts are morelikely to experience a post release bug than those reviewed onunicast. This is probably due to the good traceability featuresof unicast technologies such as JIRA. Therefore, Software or-

ganizations should consider adopting mature Review platformslike JIRA, since they are likely to improve the e�ectiveness ofReview activities.

For RQ3, we address E�ciency. Even though the reviewlength seems shorter on broadcasts in comparison to unicast,the di�erence is not statistically significant. In the case ofresponse delay, the only systems for which we found a sta-tistically significant di�erence are PIG and HBase. However,these two projects had the largest amount of patches amongour studied projects, see Figure 8, which may explain whywe observed a statistically significant di�erence. Typically, wesee that the only di�erence between the broadcast and unicasttechnology is how reviewers get patches.

Going back to the selection of the projects for this study,all five projects were at their maturity state, therefore we can’tattribute any threat to validity to this aspect. Moreover, the sizeof patches reviewed under the broadcast and the size of patchesreviewed using unicast are statistically the same. Therefore, wecan’t claim that the results were a�ected by either the projectstate or any di�erence in size3.

Analyzing Table II, we observe that the p-values calculatedfor the seven variables are statistically significant for fivemetrics (RL, NV, RQs, MRR, and RD).

Moreover, this findings are useful for our analysis because,just as we mentioned on size above, the number of reviewers isanother variable we should pay close attention to. For example,if the number of reviewers on broadcast had a statisticallysignificant di�erence with those on unicast, then it could turnout that, the result of our findings were influenced by thenumbers of reviewers. That is, we could have concluded thatmaybe there were more reviewers on the broadcast that iswhy it took less among of time to review codes than unicast.Whereas, it is not the case in this study. However, since thisis not the case, the result of our findings are very likely solelydue to the medium on which code review is performed.

Furthermore, let us revisit the survival curve in Figure 6, wenotice large steps in the staircase, which suggest that the shapeof the survival curves are strongly a�ected by minor changesin the data, i.e., one reviewer performing an additional review.

Notwithstanding, since the numbers of reviewers are not(statistically) significantly di�erent, we observe large step inthe staircases in both the broadcast and unicast. Also, in TableIV we see that unicast reviewers on average, survive morethan reviewers on broadcast. This confirms our finding thatreview done on unicast undergo more rounds than those doneon broadcast.

Additionally, the more a patch undergo rounds of reviews,the less likely it can be bug prone, this explains why applyingthe SZZ algorithm on both medium shows unicast to be moree�ective than broadcast.

Based on our findings and the Apache reviewers experts’opinion, switching from broadcast to unicast was necessary,because most of the reviewers found unicast easier to use andmore useful for code review.

3Size of patches for the projects comparing size using broadcast againstsize using unicast, see Figure 8.

Also, changing from broadcast to unicast, we thought couldhave caused delay for the reviewers (spending additional timeto learn unicast), because we were expecting that the learningcurve could have influenced the reviewing process.

Overall, our findings are consistent with the responsesgiven by Apache’s reviewers. In RQ3, our results suggestthat, unicast technology is more e�ective in catching bugsthan broadcast technology and reviewers advocate that, unicastmakes it easier for them to leave comments and visualize theexact changes that have been performed. Two aspects thatare important during a bug fixing process. Reviewers alsomentioned that the tra�c on unicast technologies, which islower in comparison to the tra�c on broadcast technologies;resulting in less distractions during the bug fixing process.

We captured a conversation on the broadcast technologyof HBase (among some reviewers) centered on the transitionfrom broadcast to unicast. In this particular conversation, thereviewer was asking in the forum if the switch has alreadytaken place. At that point we could not tell if unicast would bebetter than broadcast. However, after our analysis and feedbackfrom selected reviewers, we now conclude that it was a justifiedanxiousness.

From the results of this study, we can comment thatApache’s decision to switch to unicast for code review wasgood and justified.

V. R������ W���Several research works have been carried out in the past on

code review, in this section, we present the main contributionsthat are relevant to this paper. We identify three main areas ofcontribution that are closely related to our work.

Code review quality.Kononenko et al. investigated the quality of code review onMozilla OSS project [28]. They argue that in practice, theprocess of executing code review can still allow bugs in tothe code database or repository, their result confirms this factwith a 54% of reviewed code that were found to be buggy [28].This also agrees with our result on broadcasts as compared tounicast. To investigate the quality of code review, they used twometrics that we used in our study, the proportion of reviewedpatches that experienced a bug and reviewers participation.

Moreover, Morales et al. studied the impact of code reviewon design quality [39]. Using three projects: Qt, VTKKand ITK, they focused more on anti-pattern, and from theirfindings, they also suggest that strong code review participationpositively impacts software design quality.

Code review practice.Belli and Crisan [40], proposed an approach to improve thee�ciency of code review activities. To evaluate their proposedapproach, the authors subdivided reviewers into two indepen-dent groups. Each group analyzing the same block of codeusing two di�erent techniques; one group using their proposedapproach and the other group using the traditional manualapproach. Reviewers using their proposed approach performedshorter review sessions in comparison to those following thetraditional manual technique.

Bavota et al. show that there is a relationship between codereview quality and code review practice [29]. Their resultsreveal that codes that are reviewed before committed reducesthe likelihood of inducing bug fixes by 50%. Moreover, theyalso show that the readability of codes that are committed afterbeing reviewed is significantly higher than the readability ofunreviewed codes. They studied the usefulness of the numbersof reviewers as a factor to improve the quality of a code review.In our study, we also computed this metric and obtained similarresults that confirm their work. Moreover, their work is basedon three open source projects that use the Gerrit code reviewplatform, unlike our studied projects which uses unicast andbroadcast technologies. They also use the SZZ [33] algorithmand their result confirms our findings on the role of reviewmedia on e�ectiveness.

A more related work is the empirical study by Bettenburg etal. [23] who studied developers contributions to two systemsAndroid (Profit) and Linux Kernel (non-profit) systems. Thiswork inspired our approach of using the broadcast and unicasttechnologies to study tool-based code review [16].

Code review medium.De Alwis et al. highlight the importance of changing versioncontrol systems, in this context they show that more andmore open and closed source projects are switching fromcentralized version control systems (CVCS) to decentralizedversion control systems (DVCS) [41]. In their work, theypoints out the main di�erences between CVCS and DVCS, andexplained the rationales behind the transition and some keybenefits for projects; giving reasons for the transition. Theyalso capture some interesting discussions among developersdiscussing about the transition, in our work, we also capturedsome reviewers discussion on the transition from broadcastto unicast. Moreover, they points out major weaknesses ofthe CVCS and show how DVCS will benefit non-committers.Also, they identify the following projects, which are switchingto DVCS: (Perl, OpenO�ce, NetBSD, Python. However, theirwork was limited to what projects team members believe tobe the impact of switching to a DVCS.

VI. T������ �� V�������In this section, we discuss the limitations of our work

following common guidelines for empirical studies [42].

A. Construct validity threatsRelate to the meaningfulness of our measurement results,

in other words, this threat is solely due to errors in themeasurement of metrics. To compute our metrics, we usedMailminer2 [26] that converts the .mbox files into postgressqlrelational Database records, email threads reconstruction canbe problematic, for example some emails may contain specialcharacters that causes some of the threads not to be convertedto the database record; the risk for data loss is much largerin broadcast than unicast. Furthermore, unicast dataset is morerecent than the broadcast dataset, as a consequence, any di�er-ences observed could be due to cultural changes in the projectsrather than the Review infrastructure used. Additionally, since

di�erent kinds of projects may have di�erent complexity, itmay be a better idea to use di�erent kinds of projects withvarying complexity to compare these two review technologies.

B. Threats to internal validityRelate to alternative explanations to describe our findings.

The implementation of the SZZ used can be a threat to thisstudy, moreover, our design window can also be a potentialthreat because not all the projects switched at the same time.That is, inappropriate choice of the time window sizes mighthave lead to ecological inference fallacy [43]. The choice ofmetrics can be a threat as well to this study. However, weselected well known metrics from the code review literatureto mitigate this threat. Also, for a trade-o� between eithercomparing broadcast and unicast in di�erent systems at thesame time (no pre/post issue, but so many other confoundingfactors and noise), or comparing in the same system before andafter (closest to have minimal confounding factors, althoughnot perfect.) We selected old projects, which used the broad-cast for reviewing patches and then switched from broadcastto unicast.

C. Conclusion validity threatsConcerns the relation between the treatment and its out-

come. To address this, we were attentive not to violate theassumptions of our null hypothesis. We claim correlation andnot causality.

D. Reliability validity threatsReplication is important in science, this treat addresses how

this work can be replicated. To this e�ect, we attempt to pro-vide necessary details to replicate our study. The repositoriesand tools we used in this study are open source.

VII. C���������In this paper, we empirically studied review data of five

Apache projects that switched from broadcasts code reviewto unicast code review environment; a web-based tool. Ourobjective was to understand the impact of review technol-ogy on review e�ectiveness and quality. Our results suggestthat broadcasts reviews are twice faster than unicast reviews.However, unicast’s review quality outperforms that of broad-cast. Additionally, using the SZZ algorithm to track reviewedpatches that experienced future bugs, we observed that patchesreviewed through broadcasts are more likely to experience apost release bug than those reviewed on unicast.

Our findings suggest that the medium used for code reviewcan a�ect the e�ectiveness and hence the quality of reviewactivities. Software organizations should consider adoptingmature Review platforms like unicast platforms, since they arelikely to improve the e�ectiveness of Review activities.

R���������[1] P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida, “Investigating

code review practices in defective files: An empirical study of the qtsystem,” in Mining Software Repositories (MSR), 2015 IEEE/ACM 12thWorking Conference on, May 2015, pp. 168–179.

[2] Y. Wang, X. Zhang, L. Yu, and H. Huang, “Quality assurance of peercode review process: A web-based mis,” in Computer Science andSoftware Engineering, 2008 International Conference on, vol. 2, Dec2008, pp. 631–634.

[3] Y. Tymchuk, “Treating software quality as a first-class entity,” inSoftware Maintenance and Evolution (ICSME), 2015 IEEE InternationalConference on, Sept 2015, pp. 594–597.

[4] M. E. Fagan, “Design and code inspections to reduce errors in programdevelopment,” IBM Systems Journal, vol. 15, no. 3, pp. 182–211, 1976.

[5] M. Beller, A. Bacchelli, A. Zaidman, and E. Juergens, “Modern codereviews in open-source projects: Which problems do they fix?” inProceedings of the 11th Working Conference on Mining SoftwareRepositories, ser. MSR 2014, 2014, pp. 202–211.

[6] B. Meyer, “Design and code reviews in the age of the internet,” Commun.ACM, vol. 51, no. 9, pp. 66–71, Sep. 2008.

[7] A. Bosu, M. Greiler, and C. Bird, “Characteristics of useful codereviews: An empirical study at microsoft,” in Proceedings of the 12thWorking Conference on Mining Software Repositories, ser. MSR ’15,2015, pp. 146–156.

[8] V. Balachandran, “Fix-it: An extensible code auto-fix component inreview bot,” in Source Code Analysis and Manipulation (SCAM), 2013IEEE 13th International Working Conference on, Sept 2013, pp. 167–172.

[9] V. Balachandran, “Reducing human e�ort and improving quality in peercode reviews using automatic static analysis and reviewer recommenda-tion,” in Proceedings of the 2013 International Conference on SoftwareEngineering, ser. ICSE ’13, 2013, pp. 931–940.

[10] Y. Jiang, B. Adams, F. Khomh, and D. M. German, “Tracing back thehistory of commits in low-tech reviewing environments: A case studyof the linux kernel,” in Proceedings of the 8th ACM/IEEE InternationalSymposium on Empirical Software Engineering and Measurement, ser.ESEM ’14. New York, NY, USA: ACM, 2014, pp. 51:1–51:10.[Online]. Available: http://doi.acm.org/10.1145/2652524.2652542

[11] N. Bettenburg, E. Shihab, and A. E. Hassan, “An empirical study on therisks of using o�-the-shelf techniques for processing mailing list data,”in ICSM’09: Proceedings of the 25th IEEE International Conference onSoftware Maintenance. IEEE Computer Society, 2009, pp. 539–542.

[12] P. Thongtanunam, X. Yang, N. Yoshida, R. G. Kula, A. E. C. Cruz,K. Fujiwara, and H. Iida, “Reda: A web-based visualization tool foranalyzing modern code review dataset,” in Software Maintenance andEvolution (ICSME), 2014 IEEE International Conference on, Sept 2014,pp. 605–608.

[13] A. Guzzi, A. Bacchelli, M. Lanza, M. Pinzger, and A. v. Deursen,“Communication in open source software development mailing lists,”in Proceedings of the 10th Working Conference on Mining SoftwareRepositories, ser. MSR ’13, 2013, pp. 277–286.

[14] P. C. Rigby and A. E. Hassan, “What can oss mailing lists tellus? a preliminary psychometric text analysis of the apache developermailing list,” in Proceedings of the Fourth International Workshop onMining Software Repositories, ser. MSR ’07, 2007, pp. 23–. [Online].Available: http://dx.doi.org/10.1109/MSR.2007.35

[15] P. Rigby, B. Cleary, F. Painchaud, M. A. Storey, and D. German,“Contemporary peer review in action: Lessons from open source de-velopment,” IEEE Software, vol. 29, no. 6, pp. 56–61, Nov 2012.

[16] A. Bacchelli and C. Bird, “Expectations, outcomes, and challengesof modern code review,” in Proceedings of the 2013 InternationalConference on Software Engineering, ser. ICSE ’13, 2013, pp. 712–721.

[17] M. Squire, “Should we move to stack overflow? measuring the utilityof social media for developer support,” in 2015 IEEE/ACM 37th IEEEInternational Conference on Software Engineering, vol. 2, May 2015,pp. 219–228.

[18] V. J. Hellendoorn, P. T. Devanbu, and A. Bacchelli, “Will they like this?:Evaluating code contributions with language models,” in Proceedings ofthe 12th Working Conference on Mining Software Repositories, ser. MSR’15, 2015, pp. 157–167.

[19] C. Williams and J. Spacco, “Szz revisited: verifying when changesinduce fixes,” in Proceedings of the 2008 workshop on Defects in largesoftware systems. ACM, 2008, pp. 32–36.

[20] “Replication:guide to our tools and dataset,” https://bitbucket.org/foundjem/ml-issuetk/src, accessed: 2016-03-17.

[21] P. C. Rigby and C. Bird, “Convergent contemporary software peer reviewpractices,” in Proceedings of the 2013 9th Joint Meeting on Foundationsof Software Engineering, ser. ESEC/FSE 2013, 2013, pp. 202–212.

[22] P. C. Rigby, D. M. German, and M.-A. Storey, “Open source softwarepeer review practices: a case study of the apache server,” in Proceedingsof the 30th international conference on Software engineering. ACM,2008, pp. 541–550.

[23] N. Bettenburg, A. E. Hassan, B. Adams, and D. M. German,“Management of community contributions,” Empirical SoftwareEngineering, vol. 20, no. 1, pp. 252–289, 2013. [Online]. Available:http://dx.doi.org/10.1007/s10664-013-9284-6

[24] “Apache-Software-Foundation developer mailing list,” http://mail-archives.apache.org/mod_mbox/hbase-dev/, accessed: 2016-03-17.

[25] “Apache-Software-Foundation jira issue tracking system,” https://issues.apache.org/jira/secure/BrowseProjects.jspa#all, accessed: 2016-03-17.

[26] N. Bettenburg, E. Shihab, and A. E. Hassan, “An empirical study onthe risks of using o�-the-shelf techniques for processing mailing listdata,” in Software Maintenance, 2009. ICSM 2009. IEEE InternationalConference on, Sept 2009, pp. 539–542.

[27] “Apache-Software-Foundation github mirrior site,” https://github.com/apache, accessed: 2016-03-17.

[28] O. Kononenko, O. Baysal, L. Guerrouj, Y. Cao, and M. W. Godfrey, “In-vestigating code review quality: Do people and participation matter?” inSoftware Maintenance and Evolution (ICSME), 2015 IEEE InternationalConference on, Sept 2015, pp. 111–120.

[29] G. Bavota and B. Russo, “Four eyes are better than two: On the impactof code reviews on software quality,” in Software Maintenance andEvolution (ICSME), 2015 IEEE International Conference on, Sept 2015,pp. 81–90.

[30] C. F. Kemerer and M. C. Paulk, “The impact of design and code reviewson software quality: An empirical study based on psp data,” IEEE Trans.Softw. Eng., vol. 35, no. 4, pp. 534–550, Jul. 2009.

[31] “Apache-Software-Foundation developer mailing list,” https://smartbear.com/learn/code-review/best-practices-for-peer-code-review/, accessed:2016-03-17.

[32] D. B. Bisant and J. R. Lyle, “A two-person inspection method toimprove prog ramming productivity,” IEEE Transactions on SoftwareEngineering, vol. 15, no. 10, pp. 1294–1304, Oct 1989.

[33] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim, “Extractingstructural information from bug reports,” in Proceedings of the 2008 in-ternational working conference on Mining software repositories. ACM,2008, pp. 27–30.

[34] A. Meneely, H. Srinivasan, A. Musa, A. Rodriguez Tejeda, M. Mokary,and B. Spates, “When a patch goes bad: Exploring the properties ofvulnerability-contributing commits,” in Empirical Software Engineeringand Measurement, 2013 ACM/IEEE International Symposium on. IEEE,2013, pp. 65–74.

[35] G. Macbeth, E. Razumiejczyk, and R. D. Ledesma, “Cli�’s deltacalculator: A non-parametric e�ect size program for two groups ofobservations,” Universitas Psychologica, vol. 10, no. 2, pp. 545–555,2011.

[36] M. Mukaka, “A guide to appropriate use of correlation coe�cient inmedical research,” Malawi Medical Journal, vol. 24, no. 3, pp. 69–71,2012.

[37] O. Baysal, O. Kononen, R. Holmes, and M. W. Godfrey, “The influenceof non-technical factors on code review,” in 2013 20th Working Confer-ence on Reverse Engineering (WCRE). IEEE, 2013, pp. 122–131.

[38] B. D. Sethanandha, “Improving open source software patch contributionprocess: methods and tools,” in Software Engineering (ICSE), 2011 33rdInternational Conference on, May 2011, pp. 1134–1135.

[39] R. Morales, S. McIntosh, and F. Khomh, “Do code review practicesimpact design quality? a case study of the qt, vtk, and itk projects,” inSoftware Analysis, Evolution and Reengineering (SANER), 2015 IEEE22nd International Conference on, March 2015, pp. 171–180.

[40] F. Belli and R. Crisan, “Empirical performance analysis of computer-supported code-reviews,” in Proceedings The Eighth International Sym-posium on Software Reliability Engineering, Nov 1997, pp. 245–255.

[41] B. de Alwis and J. Sillito, “Why are software projects moving fromcentralized to decentralized version control systems?” in Cooperativeand Human Aspects on Software Engineering, 2009. CHASE ’09. ICSEWorkshop on, May 2009, pp. 36–39.

[42] R. K. Yin, Case study research: Design and methods. Sage publications,2013.

[43] D. Posnett, V. Filkov, and P. Devanbu, “Ecological inference in empiricalsoftware engineering,” in Automated Software Engineering (ASE), 201126th IEEE/ACM International Conference on, Nov 2011, pp. 362–371.