IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 58, NO. 4, AUGUST 2011 1723

DAQ Architecture Design of Daya Bay ReactorNeutrino ExperimentFei Li, Xiaolu Ji, Xiaonan Li, and Kejun Zhu

Abstract—The Daya Bay Reactor Neutrino Experiment consistsof seventeen detectors distributed in three underground experi-mental halls. Each detector has a separate VME readout crate thatcontains the trigger and electronics modules. The data acquisition(DAQ) system reads data fragments from electronics and triggermodules and concatenates them into an event in each crate. TheDAQ monitors the data quality, merges the event streams fromeach hall, and records these data to disk. The DAQ architectureis designed as a multi-level system based on the BESIII DAQ andATLAS TDAQ systems using embedded Linux, VME bus system,advanced commercial computers and distributed network tech-nologies. This paper presents the main DAQ design requirements,and the hardware and software architectures.

Index Terms—Data acquisition systems, distributed computing,high energy physics instrumentation computing, real time systems.

I. INTRODUCTION

T HE Daya Bay Reactor Neutrino Experiment is a neutrino-oscillation experiment designed to measure the mixing

angle using anti-neutrinos produced by the reactors of theDaya Bay Nuclear Power Plant (NPP) and the Ling Ao NPPin Shenzhen, China. The basic experimental layout consists ofthree underground experimental halls, one far and two near,linked by horizontal tunnels as shown in Fig. 1. The distancesbetween Daya Bay(DB) near, Ling Ao(LA) near and Far hallswith surface tunnel entrance are about 300, 1600 and 2000 me-ters [1].

Eight identical cylindrical anti-neutrino detectors (AD) willbe deployed as shown, with 192 photomultiplier tubes (PMT)in each. The ADs are deployed within water pools designedto reduce background radiation and are instrumented to detectcosmic rays. Segmented as two separate water Cherenkov de-tectors, the inner and outer water shield detectors contain be-tween 123 and 212 PMTs that surround the ADs. A resistiveplate chamber (RPC) detector covers each water pool. Overallthere are seventeen independent detectors in the Daya Bay ex-periment (eight ADs, six water shields, and three RPC detec-tors). Each detector will be readout using an independent VMEcrate that contains both trigger and detector readout electronics.A custom distributed clock system provides a synchronized 10MHz clock and an absolute time stamp for triggered events.

Manuscript received June 14, 2010; revised December 01, 2010, February 18,2011, and April 08, 2011; accepted April 16, 2011. Date of publication July 12,2011; date of current version August 17, 2011. This work was supported by theMinistry of Science and Technology of China (No. 2006CB808103).

The authors are with the Institute of High Energy Physics, Chinese Academyof Sciences, 100049 Beijing, China (e-mail: lifei@ihep.ac.cn).

Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TNS.2011.2158657

Fig. 1. Default configuration of the Daya Bay experiment.

The Daya Bay data acquisition (DAQ) system receives inter-rupts from the trigger module in each crate, reads-out data frag-ments from all modules resident in the crate with chain blocktransfer (CBLT) mode, and concatenates them into an event,crate by crate in parallel. Then DAQ transmits the data to backend to do event monitoring and event stream merging which or-ders the events using the trigger time stamp. The DAQ systemalso provides an interactive interface to the electronics, triggerand calibration systems. The DAQ system design impacts thecapabilities and dependability of the overall integrated experi-ment.

II. SYSTEM REQUIREMENTS

A. Run Control Requirement

Since the detectors are independent, the experiment requiresmultiple independent DAQ systems in each hall. The three ex-perimental halls are far away from the surface control roomwhich is located outside of tunnel. The run control should beconfigurable and flexible, allowing it to work seamlessly fordata taking controlled from the surface or in a detector hall. Therun control should allow both global operation of all detectorsystems in all three detector-halls, and operation of sub-sets ofdetectors whenever debugging or commissioning is required.The system should also allow for multiple run-control opera-tions in parallel in any detector hall so that debugging and com-missioning can be conducted simultaneously in more than onelocation.

0018-9499/$26.00 © 2011 British Crown Copyright

1724 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 58, NO. 4, AUGUST 2011

TABLE ISUMMARY OF RUN MODE REQUIREMENTS

B. Run Mode Requirements

The DAQ should be capable of taking different data typesfrom detectors running together in different run modes. Table Ishows the summary of run mode requirements. The DAQ alsoneeds to automatically control data taking, assort and exchangeparameters with external interactive systems during calibrationand monitoring running modes.

C. Data Readout and Processing Requirements

Event building is executed within each VME crate. There aretwo types of VME detector readout electronics, one for PMTs(ADs and water shield detectors) and another for the RPC de-tectors. All readout electronic modules are custom modules de-signed and manufactured by engineers of the Daya Bay collab-oration, requiring special integration and debugging with theDAQ.

All data created within a hall should be merged. Mergingshould be configurable for any combination of detectors anddifferent run modes. Furthermore, the events should be sortedin the merged stream by the time stamp read from the triggermodule. A policy to name the merged stream is needed. Finally,event data must be recorded to a disk array.

Raw event data must be processed and analyzed online forreal time monitoring of the detectors and readout systems.

D. Throughput Requirements

The expected maximum physics data throughput rate is lessthan 500 KB/s for one crate and less than 1.5 MB/s for one hallassuming a baseline trigger rate. The total normal physics datathroughput rate for all 3 halls is expected to be about 3 MB/s[1]. These estimates would increase with implementation of fullwaveform digitization, noisy PMTs, or implementation of addi-tional triggers (e.g., LED calibration triggers). The system de-sign should have sufficient flexibility to allow for backgroundstudies or noise research. Consequently, the design DAQ eventrate could reach 1 kHz with a 2 kilobyte event size resulting ina throughput less than 2 MB/s per PMT detector crate. As such,the overall DAQ must be capable of managing a total experi-mental hall throughput of up to 35 Mbytes/s.

The DAQ system is also required to have a negligible readoutdead-time. This requires fast online memory buffers that canhold multiple detector readout snapshots while the highest levelDAQ CPUs perform online processing and transfer to perma-nent storage.

E. Other Common Requirements

Hardware parameters, electronic layout and software setupshould be configurable. The DAQ also needs to provide a graph-ical user interface to display the running status and for moni-toring using histograms. Bookkeeping should include run con-figuration, run parameters, run summary, running status, errorsand logs. Finally, the experimental site is far away from mostcollaboration members such that remote operations should bepossible with good stability and robustness.

III. ARCHITECTURE AND HARDWARE SYSTEM DESIGN

Based on construction experience for BESIII DAQ [2], [3],the Daya Bay DAQ architecture is designed as a multi-levelsystem using embedded Linux, advanced commercial com-puters and distributed network technology. The hardware andnetwork system had a united design and common devices withdetector control and offline systems. The onsite computingsystem is shown in Fig. 2. The system has been designedentirely with a gigabit ethernet network. Due to the distancesinvolved, the three experimental halls connect to the surfacethrough single mode fibers. Double peer to peer single modefiber cables connect the front-end and back-end networks.Double switches will be placed in each hall for redundancy andreliability.

The DAQ system can be separated into two parts: VMEfront-end system and back-end system. The front-end readoutsystem is a real-time embedded system based on a VME bus.Each VME crate holds a VME system controller MVME5500, an embedded single-board computer manufactured byMotorola. It is based on a PowerPC CPU and Universe II chipfor VME bus interface. The operating system running on thecontroller is TimeSys, a commercial embedded real-time Linuxwith version 2.6.9 kernel. The system throughput can exceed2.5 kHz with a 2 kilobyte event size, which is sufficient to meetthe experimental requirements [4].

All computers except a local control PC are placed in thesurface computer room for more convenient management andmaintenance. The Daya Bay experiment also adopted the bladeserver based computing farm to construct the back end system.Computing farms are widely used in the field of high-energyphysics. Blade server technology greatly increases server den-sity, lowers power requirements and cooling costs, eases serverexpansion and simplifies data center management [5]. The DAQhas two x3650 servers acting as files servers for farm and datastorage. Nine blade servers serve as computing nodes for datagathering and data quality monitoring.

IV. SOFTWARE ARCHITECTURE DESIGN

The DAQ system needs a distributed software solution tomeet above requirements and architecture design. ATLASTDAQ software implements a CORBA based inter-processcommunication framework [6]–[8]. And BESIII DAQ softwareis ported to the framework [9]. The Daya Bay DAQ software isdesigned and developed based on the ATLAS TDAQ softwareand BESIII DAQ software. Functionally, the DAQ softwarecan be divided into two layers as shown in Fig. 3: the dataflow software and the online software. The data flow softwareis responsible for all the processing of physics data, including

LI et al.: DAQ ARCHITECTURE DESIGN OF DAYA BAY REACTOR NEUTRINO EXPERIMENT 1725

Fig. 2. Architecture of DAQ.

Fig. 3. Software architecture of DAQ.

receiving and transporting the data to storage. The onlinesoftware is responsible for all aspects of experimental and DAQoperations and controls during data-taking. The online softwareprovides services to data flow.

A. Online Software Design

The Online Software is customized from the ATLAS onlineframework and migrated to a TimeSys Linux/PowerPC em-bedded system environment. The ATLAS Online Software isthe global system software of the DAQ system to configure,control, and share information. It is able to start, stop andsynchronize the order of all programs on all processors. It isa distributed software framework based on CORBA. It pro-vides essentially the ‘glue’ that holds the various sub-systemstogether. It does not contain any elements that are detectorspecific as it is used by all the various configurations of theDAQ and detector instrumentation. It also provides the interfacebetween the human user and the DAQ system.

The Online Software architecture is based on a componentmodel and consists of three high level components, called pack-ages.

Control contains sub-packages for the control of theDAQ system detectors. Control sub-packages exist to sup-

port DAQ system’s initialization and shutdown, to pro-vide control command distribution, synchronization, errorhandling, and system verification. The Process Manager(PMG) provides the basic process management function-ality in a distributed environment. It starts, stops and mon-itors processes on the different DAQ hosts. To execute therequests, the PMG runs agents on all the machines and usesthe information provided by the user and/or the configura-tion databases. A Graphical User Interface (GUI) allowsthe user to send commands to control as well as to monitorthe system.

Databases contains sub-packages for configuration ofthe DAQ system and detectors. Configuration sub-pack-ages exist to support the system configuration descriptionand access to it, record operational information during arun and access to this information.

Information Sharing contains classes to support infor-mation sharing in the DAQ system. Information Sharingclasses exist to report error messages, to publish states andstatistics, to distribute histograms built by the sub-systemsof the DAQ system and detectors, and to distribute eventssampled from different parts of the experiment’s data flowchain.

The interaction between the Online Software packages isshown in Fig. 4. The Control makes use of the InformationSharing and of the Databases packages. The Databases packageis used to describe the system to be controlled. The InformationSharing package provides the infrastructure to obtain andpublish information on the status of the controlled system, toreport and receive error messages, and to publish results forinteraction with the operator [10].

1726 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 58, NO. 4, AUGUST 2011

Fig. 4. Interaction diagram of online component.

Fig. 5. Data flow component diagram.

Fig. 6. Deployment design diagram.

B. Data Flow Software Design

Fig. 5 shows the data flow components diagram of the DayaBay DAQ. It was migrated from the ATLAS TDAQ backenddata flow software and BESIII front readout software [11]–[15].The read out system (ROS) runs on a Timesys Linux/PowerPC.One ROS runs on one MVME 5500 controller. It reads data,packs events and sends the data to the back-end.

Other components run on a SLC4/X86. The Event flow dis-tributor (EFD) can receive data from multiple ROSs through aninput task. The linked ROS data will be merged together thensent to sub farm output (SFO) by EFD. The event flow inputoutput (EFIO) service package is used to deal with data trans-portation among the data flow component.

The monitoring task of the EFD will parse sent data and filldata quality monitoring histograms, then publish the data to aninformation sharing server. The external task of the EFD canshare the event data with an external process task (PT). PT isused for an option to do further data processing or analysis. Dataanalysis will require significant CPU resource such that EFDand PT should be run on blade servers.

SFO can also receive multi EFDs’ data, merge and sort eventsby trigger time, then record to data files. The SFO must run onfile servers for better storage performance with locally mounteddisks.

C. System Deployment Design

The DAQ system can be separated into multiple partitionsto minimize correlations among hardware and software of dif-ferent detectors. Multiple detectors can run and be controlledtogether as a partition. The participants of a partition can be con-figurable. Each detector can be an individual partition by itself.Partitions which use no conflicting resources can run separately.Each partition is independent of each other logically. Each par-tition has its own hardware and software resource and config-uration file. Run control and communications are only carriedout inside a partition. Parallel running depends on different par-tition configuration files.

A typical partition deployment design is shown in Fig. 6. EachROS corresponds to a VME crate and the detector. The wholesystem is separated into three partitions for three experimentalhalls. Each partition includes all ROS of one hall, several EFDs

LI et al.: DAQ ARCHITECTURE DESIGN OF DAYA BAY REACTOR NEUTRINO EXPERIMENT 1727

and one SFO. Several EFDs of one partition can be linked tospecific ROSs according to these ROSs’ throughput or detectortype. All event data of one hall will be recorded to one singledata file stream by a SFO.

V. SUMMARY

Most servers and the surface network have been installed andare working well, and the first two ADs have been assembled.A full integration test was conducted onsite in surface assemblybuilding. One VME crate housing thirteen electronic PMTreadout modules and one trigger module was used to readouteach of the assembled ADs separately. The DAQ operated oneMVME 5500 running one ROS, one blade server running oneEFD, one file server running one SFO. A maximum trigger rateof 1.05 kHz with a 10 kilobyte event size reading out noisesignals was achieved. The maximum trigger rate for an eventsize of 2 kilobytes was 2.8 kHz. A 72-hour stability test withan approximate 1.5 kHz event rate and 2.3 kilobyte mean eventsize was successfully completed without any crashes.

A DAQ software test for multiple VME crates is set up inlaboratory. It composes of seven software emulated ROS run-ning on MVME 5500, seven corresponding EFD running on twoblades and one SFO running on one storage server. The serversand VME crates connect by more than 2 kilometers fiber cable.The maximum event rate for an event size of 2 kilobytes canreach 8 kHz with data stream merging and time sorting now.Each ROS-EFD-SFO data flow link can handle above 1.1 kHzevent rate on average.

Based on the ATLAS TDAQ and BESIII DAQ, the architec-ture design of the Daya Bay DAQ was achieved. The singlesubsystem (one PMT detector) DAQ software was developed

rapidly with minimal manpower and reusing ATLAS and BE-SIII software. Preliminary test results show that the DAQ per-formed well and the architecture design meets current experi-mental requirements. More detail design and development areongoing. Further validation testing is planned in the coming in-stallation and commissioning phases.

REFERENCES

[1] Daya Bay Collaboration, “Daya Bay Project Technical Design Report(v2),” Jan. 22, 2008.

[2] BESIII Collaboration, “The construction of the BESIII experiment,”Nucl. Instr. Meth. A, vol. 598, no. 1, pp. 7–11, Jan. 2009.

[3] M. Ablikim et al., “Design and construction of the BESIII detector,”Nucl. Instr. Meth. A, vol. 614, no. 3, pp. 345–399, Mar. 11, 2010.

[4] X. Ji et al., “Research and design of DAQ system for Daya Bay re-actor neutrino experiment,” in Proc. IEEE Nuclear Science Symp. Conf.Record, Dresden, Germany, 2008, pp. 2119–2121, N30-76.

[5] F. Li et al., “Design and implementation of BESIII online farm,” inProc. IEEE Nuclear Science Symp. Conf. Record, Dresden, Germany,2008, pp. 2112–2113, N30-72.

[6] R. Jones et al., “Applications of CORBA in the ATLAS prototypeDAQ,” IEEE Trans. Nucl. Sci, vol. 47, pp. 331–335, Apr. 2000.

[7] ATLAS DAQ Backend Software Group, “ATLAS DAQ Back-EndSoftware High-Level Design,” 1998.

[8] ATLAS TDAQ group, “ATLAS High-Level Trigger, Data Acquisitionand Controls Technical Design Report,” 2003.

[9] F. Li et al., “Distributed inter process communication framework ofBESIII DAQ online software,” (in Chinese) Nucl. Electron. DetectionTechnol., vol. 26, no. 5, pp. 605–608, Sep. 2006.

[10] I. Alexandrov et al., “Online software for the ATLAS test beam dataacquisition system,” IEEE Trans. Nucl. Sci., vol. 51, no. 3, Jun. 2004.

[11] H. P. Beck et al., “The base line data flow system of the ATLAS triggerand DAQ,” IEEE Trans. Nucl. Sci., vol. 51, pp. 470–475, Jun. 2004.

[12] C. Haeberli et al., “ATLAS TDAQ data collection software,” IEEETrans. Nucl. Sci., vol. 51, no. 3, pp. 578–584, Jun. 2004.

[13] H. Zhu et al., “The implementation and application of statecharts inreal-time system in high energy physics,” IEEE Trans. Nucl. Sci., vol.53, no. 3, pp. 1032–1038, Jun. 2006.

[14] F. Li et al., “Online data processing and analyzing in BESIII DAQ,” inProc. 16th IEEE-NPSS Real Time Conf. Record, Beijing, China, 2009,pp. 458–460.

[15] Q.-X. Zhang et al., “The control of data flow in event building of BESIIIDAQ software,” (in Chinese) Nucl. Electron. Detection Technol., vol.26, no. 3, pp. 302–306, May 2006.