There was a time when humanity faced theuniverse alone and without a friend. Nowhe has creatures to help him; stronger crea-tures than himself more faithful, more use-ful, and absolutely devoted to him. Mankindis no longer alone.

—Isaac Asimov, I, Robot

Almost all software engineering ex-perts agree that continuous integra-tion is superior to the “big-bang” in-tegration approach employed in the

dark ages. Especially agile methodologiessuch as Extreme Programming (XP) pro-mote development processes that let de-velopers add functionality little-by-little,thereby significantly reducing project risk.

To ensure that frequent code check-ins (so-called “code deliveries,” or “de-liveries,” for short) are of high quality,frequent build and test cycles are in-evitable. One such approach is the nowfamous “Daily Build and Smoke Test”process, described in detail in Steve Mc-Connell’s book Rapid Development (Mi-crosoft Press, 1996). While daily buildingand smoke testing significantly reducesproject risk, for large and complex pro-jects you may want to take this approachto extremes by validating your code baseon a check-in basis— that is, many timesa day. The advantages are obvious. If youcheck in buggy code, it will be immedi-ately detected, which means that there

will be no more time-consuming “Who’sbroken the build?” quests.

In this article, I present an approachthat puts the Herculean building and test-ing effort on the shoulders of a test robot,or “testbot.”

The ChallengeOne of my current projects is a complexembedded-systems project. It isn’t partic-ularly large (around 20 developers and 100KLOC), but it is a multiplatform, multifea-ture kind of project. At any time— or moreprecisely, within a rather short period oftime— the system has to be in a releasablestate for any hardware platform and vari-ous combinations of features supportedby our common code base. Here, I referto such a combination of features on a par-ticular hardware platform as a “target.”

Working on this project used to be likewalking on a mine field. If you added afeature or fixed a bug for some target,there was a certain probability that youbroke the build on another target or atleast introduced a compiler warning. Overtime, these issues tended to accumulate,resulting in a downward spiral. Lots of ef-fort had to be expended later on to getback to a clean build.

Obviously, requiring the developers toverify all of the other platforms and prod-ucts before delivering their changes wouldhave taken way too much time, let alonethe fact that not every developer had ac-cess to all of the toolchains (compilers,linkers, in-circuit emulators, and the like)that the code base supports.

The Innards of a TestbotI solved this problem by assigning dedi-cated testbots to all the targets we sup-port. The testbots are workstations thathost a program that listens on the source-code-versioning system (ClearCase, in ourcase) for new deliveries of developers.When a developer finishes delivering her

task to the integration branch, the testbotexecutes the following steps:

1. Inform developer(s). To inform you thatyour check-in is under test, an e-mailis sent to you, plus a carbon-copy tothe testbot administrator; see Figure 1.The e-mail details what is being tested(whose deliveries) and contains a linkto the testbot’s logfile. By inspecting thelogfile, you can track the status of the

testbot’s work. In order to implementsuch an e-mail notification scheme, youneed a username to e-mail addressmapping. In our case, we use a simpletwo-column text file that even lets de-velopers specify their mobile phone num-bers. This enables the testbot to send youa short message via an e-mail-to-SMSgateway in case something went wrong(or everything built and tested fine).

2. Checkout code. The testbot checks outa private copy of the source code basedon the last completed delivery.

3. Build the code. The whole code is builtand possibly— as it is the case on ourproject— checked against compiler andLint warnings.

Ralf is principal of PERA Software Solu-tions and can be contacted at rholly@pera-software.com.

“Testbots areworkstations thathost a program thatlistens on thesource-code-versioning system”

Testing, with a littlehelp from your friends

I, Testbot

E M B E D D E D S Y S T E M S

42 Dr. Dobb’s Journal, May 2006 http://www.ddj.com

RALF HOLLY

4. Execute (smoke) tests. Depending onthe size and the needs of the project,the testbot executes the tests. In ourcase, we restricted the test set to teststhat only cover the most importantnominal cases. If test execution con-sumes too much time, you might notbe able to check enough code deliver-ies per day and hence will detect de-fects too late; that is, when the devel-opers already went home.

5. Inform developer(s). The last step issending another e-mail (and/or shortmessage, if you want) to you, inform-ing you about the outcome of the test.If the testbot spots a problem, it addi-tionally sends a copy to all team mem-bers, warning them about problems onthe integration branch; see Figure 2.

The testbot software was quite easy toimplement. In total, it comprises less than1000 lines of Perl code; however, the sizedepends on the kind of source-code-versioning system you are using— dif-ferent systems offer different solutions forparallel software development and query-ing of check-in information. Listing Oneis the testbot pseudocode implementation.For simplicity, I’ve left out logging and er-ror handling.

The TX ModelA small disadvantage of the testbot I’vejust described is that it heavily relies one-mail notification. Oftentimes, develop-ers complain about the flooding of e-mailsthat the testbots send, just because somedeveloper caused a Lint warning on a tar-get they are only marginally interested in.

Thus, I crafted an extended version (seeListing Two) that differentiates betweenmajor and minor issues:

• Major issues. Major issues are problemsthat prevent other developers from in-tegrating their code. We decided thatcompile- and link-time problems, as wellas smoke test execution errors, belongto this category. Because of their sever-ity, major issues are broadcast— just likebefore— by e-mail to all developers.

• Minor issues. Unlike major issues, mi-nor issues do not prevent further codedeliveries and hence are only reportedby e-mail to the developer who deliv-ered the code, the architect, and the pro-ject lead. On our project, compiler andLint warnings belong to this category.

In addition to the e-mail-notificationmechanism, there is a client program called“Observer” (implemented in Perl/Tk) run-ning on all project member’s workstations.The Observer monitors the state of all tar-gets. The following information can be de-rived from Figure 3:

1. The last update to the screen occurredat 14:02:50 and happened due to a statechange (start of build 504) reported bythe testbot of the Venus target (indicat-ed by the asterisk next to the name).

2. Because of the delivery of JOHNC etal. (you can get the full list of devel-opers who delivered to the integrationbranch by placing your mouse cursorover “JOHNC…”), the Mars project cur-rently has major issues. The previousrun based on the check-in of PETERZ,however, was fine. Since the Mars test-bot is idling, we can assume thatJOHNC et al. haven’t delivered a fix yet;hopefully, they are working on it.

3. JOHNC et al. didn’t cause any troubleto the Jupiter target.

4. Saturn is still being tested, but the pre-vious run (build 473) reported minor is-sues. Most likely, JACKF just fixed theseissues because he is also the originatorof the current testbot run (build 474).

5. Pluto and Venus are still being tested.The previous run was fine.

To provide Observer with build and teststate information, the testbots drop cook-ies on a shared file server directory. Fig-ure 4 presents the contents of the cookiefiles for a particular target. The Observerregularly checks this shared directory for

http://www.ddj.com Dr. Dobb’s Journal, May 2006 43

Figure 2: E-mail informing the whole team about problems related to JOHNCand GARYW’s check-ins.

Figure 1: E-mail notifying developers JOHNC and GARYW that their check-insare under test.

Listing Oneloop forever

sleep for 5 minutescheck versioning system for new deliveriesif new deliveries since last testbot run

get list of developers who deliveredforeach developer who delivered

send 'test started' emailcheckout code based on last deliveryexecute buildif build problems

foreach team membersend 'build problems' email

redo loop execute smoke testif smoke test problems

foreach team membersend 'smoke test problems' email

redo loop foreach developer who delivered

send 'test successful' email redo loop

Listing Twoloop forever

sleep for 5 minutescheck versioning system for new deliveries

if new deliveries since last testbot runget list of developers who delivereddrop 'build in progress' cookie (target.current)foreach developer who delivered

send 'test started' emailcheckout code based on last deliveryexecute buildif build major problems

drop 'build major problems' cookie (target.lkb)foreach team member

send 'build problems' email redo loop if build minor problems

drop 'build minor problems' cookie (target.lkb)foreach team member, architect, project lead

send 'build minor problems' email redo loop execute smoke testif smoke test problems

drop 'smoke test problems' cookie (target.lkb)foreach team member

send 'smoke test problems' email redo loop drop 'test successful' cookie (target.lkgb)remove 'build in progress' cookie (target.current)foreach developer who delivered

send 'test successful' email redo loop

DDJ

changes and displays the information ina user-friendly dialog.

By employing this mixed push/pull (e-mail/Observer) scheme, we significantlyreduced the number of e-mail broadcasts,thus improving developer happiness.Moreover, the Observer is a nice tool forproject leads because they get constant

feedback on the quality of their and oth-er leads’ projects.

The Sky Is the LimitOnce you have build and test automation,as well as a testbot framework in place,you can assign all kinds of useful tasks toyour testbots. Why not have a testbot ex-

ecute all of your system tests, stress tests,and performance measurements manymore times than you used to? On our pro-ject, we have a testbot that takes care ofexecuting all of the thousands of devel-oper (unit) tests. It iterates over a list towhich developers add references to theirdeveloper test suites. Of course, these ex-tended tests take days to execute, imply-ing that you cannot test individual check-ins but rather whole sets of check-ins. Still,if something goes wrong, you know ex-actly who made changes to the code baseand who is likely to be the culprit.

ConclusionDelivering functionality in small incrementsis the preferred way of developing systemsthese days. Especially on medium to largeprojects, testbots help ensure code qualityand reduce project risk through frequenttesting— much more testing than any hu-man being is able to bear. Equip your test-bot with a friendly UI and it will be, as theSirius Cybernetics Corporation would put it,“your plastic pal who’s fun to be with.”

DDJ

44 Dr. Dobb’s Journal, May 2006 http://www.ddj.com

Shortly after I had finished this article,the open-source project CruiseControl(http://cruisecontrol.sourceforge.

net/) was brought to my attention. Sofar, I haven’t had a chance to use it ina project, but from what I’ve seen ontheir web site, it looks like a solid toolto me. It comes with e-mail notifica-tion support and reporting facilities. Be-fore you reinvent the (build)loop, youmight want to give CruiseControl a try.Let me know what you think of it.

—R.H.

CruiseControl

Figure 3: Data displayed by the Observer tool.

Figure 4: The contents of the Mars testbot’s cookie files.