Download - Virgo Data Acquisition D. Verkindt, LAPP
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
1
Virgo Data AcquisitionD. Verkindt, LAPP
• DAQ Purpose
• DAQ Architecture
• Data Acquisition examples
• Connection to DAQ and monitoring tools
• Data Streams
• Online analysis tools
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
2
DAQ purpose
DAQ requirements:
• collection of distributed data (timing system, optical links)
• flexibility in data flow (frame format)
• reliability (at least 1 month without crash)
• easyness of use and restart (DAQ graphical client)
DAQ requirements:
• collection of distributed data (timing system, optical links)
• flexibility in data flow (frame format)
• reliability (at least 1 month without crash)
• easyness of use and restart (DAQ graphical client)
Laser
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
3
DAQ purpose
Control BuildingCentral Building
Get data from various synchronized sources, sometimes 3 km away
North Building
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
4
Data Acquisition
DetectionEnvironment Controls
DAQ purpose
Collect distributed data from:• ITF environment• ITF controls• ITF output detection
Collect distributed data from:• ITF environment• ITF controls• ITF output detection
Env. monitoringEnv. monitoringSuspension control
Output MC BenchDetection Bench
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
5
DAQ architecture
Central data collectionCentral data collection
9 MB/s (compressed = 4MB/s)
local data collector
2.7 MB/s
frames
Input bench monitoring, Vacuum monitoring, Environment monitoring
3.3 MB/s
local data collector
Suspensions dataLocking and alignment data
frames
Environment Monitoring
3.0 MB/s
frames
local data collector
Photodiodes datadet. Bench monitoring
DetectionControls
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
6
DAQ architecture
FbF1 Gx FbF
PhotodiodesAlignement
FbS
Susp. CtrlGlobal Ctrl
6 Gx4 Fbf
Laser + env + towers+ tubes + calib.itf
FbF 3 Gx
local Main Frame Builder
Central Main Frame BuilderCentral Main Frame Builder
FbS
Det. Bench Ctrl
FbS
3.0 MB/s3.3 MB/s2.7 MB/s
9 MB/s (compressed = 4MB/s)
Environment Monitoring Controls Detection
frames framesframes
local Main Frame Builderlocal Main Frame Builder
DOLDOLDOL > 30 VME crates > 30 VME crates
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
7
DAQ architecture
More than 30 VME crates, but a reduced set of standard tools:
• Digital Optical Links (DOL) for controls• Fast Ethernet and Gbit Ethernet for central data collection
• VME crates for front-end data acquisition• Workstations for central data collection
• Standard format for data collection : frames encapsulated in Ethernet messages
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
8
Frame format
Frame = elementary time slice of dataFrame = elementary time slice of data
GW signal
channel 1
frame 1 frame 2 ...
Time
channel 2
channel 3
channel n
Contains:• GPS time stamp• ITF informations• raw data channels• processed data• events
Contains:• GPS time stamp• ITF informations• raw data channels• processed data• events
Common format of several gravitational waves detectors
Common format of several gravitational waves detectors
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
9
Timing system overview
GPSTiming
Laser
Data Acquisition
DetectionEnvironment Controls
Timing Distributor Crate User’s Timing CratesUser’s Timing Crates
ADC,DACCamera,DOL
Timing
CPU
Coax Cables
ADC,DACCamera,DOL
Timing
CPU
ADC,DACCamera,DOL
Timing
CPU
ADC,DACCamera,DOL
Timing
CPU
OPT/TTLRun
OPT/TTLFrame
OPT/TTLSampling
OPT/TTLFast Clock
TTL/OPTFrame
Sampling
Timing Distributor Crate User’s Timing CratesUser’s Timing Crates
ADC,DACCamera,DOL
Timing
CPU
Coax Cables
ADC,DACCamera,DOL
Timing
CPU
ADC,DACCamera,DOL
Timing
CPU
ADC,DACCamera,DOL
Timing
CPU
OPT/TTLRun
OPT/TTLFrame
OPT/TTLSampling
OPT/TTLFast Clock
TTL/OPTFrame
Sampling
Timing Distributor Crate User’s Timing CratesUser’s Timing Crates
ADC,DACCamera,DOL
Timing
CPU
Coax Cables
ADC,DACCamera,DOL
Timing
CPU
ADC,DACCamera,DOL
Timing
CPU
ADC,DACCamera,DOL
Timing
CPU
OPT/TTLRun
OPT/TTLFrame
OPT/TTLSampling
OPT/TTLFast Clock
TTL/OPTFrame
Sampling
Timing Distributor Crate User’s Timing CratesUser’s Timing Crates
ADC,DACCamera,DOL
Timing
CPU
Coax Cables
ADC,DACCamera,DOL
Timing
CPU
ADC,DACCamera,DOL
Timing
CPU
ADC,DACCamera,DOL
Timing
CPU
OPT/TTLRun
OPT/TTLFrame
OPT/TTLSampling
OPT/TTLFast Clock
TTL/OPTFrame
Sampling
Optical fibers
Timing Information (Cm)
all VME crates synchronized by Master clock• Fast Clock (2.5 Mhz)• Sampling (20 kHz)• Frame (1 Hz)
Monitoring & Control PartMonitoring & Control Part
Generator & Distributor PartGenerator & Distributor Part
GPS
CPU
TTL/OPTRun
TTL/OPTFrame
TTL/OPTFast Clock
Timing
TTL/OPTSampling
OPT/TTLSampling
OPT/TTLFrame
Build. Return Timing
Return GPS
GPS
Thanks to A. Masserot
Purpose• Synchronization (of controls)• Frame and sampling numbers• GPS time stamp for data exchange
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
10
SMS dataMain Frame Builder
frames
timing info
Slow Frame Builder
GPSTiming
timing info
Data acquisition examples
Slow Monitoring Stations
query
Sensor (temp. pressure…)
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
11
accelerometers, microphones, … Fast Frame Builder
BNC cables
GPSTiming
timing signals
Main Frame Builder
frames
Eth. 100 Mbps
timing info
Data acquisition examples
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
12
Fast Frame Builder
Optical line
(DOL)
Main Frame Builderframes
Eth. 100 Mbps
timing info
Data acquisition examples
Photodiode
Pre-ampli , demodulation&
filtering
Photodiode Readout
GPSTiming
timing signals
Optical lineInterferometer controls
(DOL)
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
13
Connection to DAQ
Slow Frame BuilderSlow Frame Builder
Main Frame Builder
Consumer 2Consumer 1
Fast Frame BuildersFast Frame Builders
Central Main Frame Builder
Central Main Frame Builder
Main data stream
Producer
DAQ world
Data Storage
Shared Memory
Online Processing Online Processing
Main Frame Builder:
Use shared memory and 2 processes
• Producer: merge input frames and put result in shared memory
• Consumer: read frames in shared memory and send them on network
Main Frame Builder:
Use shared memory and 2 processes
• Producer: merge input frames and put result in shared memory
• Consumer: read frames in shared memory and send them on network
dataDisplay dataDisplay
Monitoring worldrequested data request
Dynamical connection
• connect: send request with list of channels
• disconnect: automatic
• minimal perturbation on main stream.
Dynamical connection
• connect: send request with list of channels
• disconnect: automatic
• minimal perturbation on main stream.
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
14
Online Monitoring using dataDisplay
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
15
Offline use of dataDisplay
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
16
DAQ control and monitoring
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
17
Web DAQ Monitoring
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
18
DAQ Performances
nADC
nBytes
• Run almost continuously since Sept. 2001
• DAQ efficiency during last engineering runs > 99.8%
• Minimized latency --> DAQ can be used for online control
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
19
Current data streams
Raw data frames:
• most of channels sampled at 20 kHz or 10 kHz
• frame = 1 sec of raw data = 4MB (day=345 GB year=120 TB)
Raw data frames:
• most of channels sampled at 20 kHz or 10 kHz
• frame = 1 sec of raw data = 4MB (day=345 GB year=120 TB)
50Hz data frames: 3% of raw data storage
• provide fast access to raw data in low frequency band
• resampling at 50Hz (with filtering) all the fast data channels
• frame = 10 sec of resampled data = 1.1 MB (day=9 GB year=3300 GB)
50Hz data frames: 3% of raw data storage
• provide fast access to raw data in low frequency band
• resampling at 50Hz (with filtering) all the fast data channels
• frame = 10 sec of resampled data = 1.1 MB (day=9 GB year=3300 GB)
Trend data frames: 0.1% of raw data storage
• provide fast access to long (weeks) stretch of data
• trend data = min, max, mean, rms computed for each fast sampled channel, over one frame
• frame = 30mn of trend data = 9.6 MB (day=460 MB year=170 GB)
Trend data frames: 0.1% of raw data storage
• provide fast access to long (weeks) stretch of data
• trend data = min, max, mean, rms computed for each fast sampled channel, over one frame
• frame = 30mn of trend data = 9.6 MB (day=460 MB year=170 GB)
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
20
Trend data acquisition
Trend Frames Disks
Trend Frame Builder
Full FrameStorage (disks)
Main Frame Builder
ControlsFrame Builder
DetectionFrame Builder
Env. MoniFrame Builder
Vega DB
(Root)
Web
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
21
Online Monitoring using trend data
Example 1 : output of ITF over 8 hours, during engineering run E4 (min, max, mean)
Use of Vega tool
and Web browser
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
22
Offline use of trend dataExample 2 : max of output of ITF, building temp. and seismic motion near north tower over 3 days
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
23
50Hz data acquisition
Trend Frames Disks
Trend Frame Builder
Full FrameStorage (disks)
Main Frame Builder
ControlsFrame Builder
DetectionFrame Builder
Env. MoniFrame Builder
50Hz Frames Disks
50 HzFrame Builder
50Hz processing
50Hz processing
50Hz processing
Vega DB
(Root)
Web
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
24
Online monitoring using 50 Hz dataExample 1 : monitoring of seismic activity over 8 hours, in 3 frequency bands
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
25
Offline use of 50 Hz data
Example 2 : spectral density of output of ITF over 3 hours of data (made in 30 sec)
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
26
Online analysis tools
GAI (General Algorithm Interface):
• A software tool to interface algorithms to online processing stream of data.
• Used to run online algorithms during engineering runs
• Used also offline to analyse engineering runs data
• Improved thanks to requests and comments from users and algorithm developers
Some of the algorithms developed up to now with GAI for online and offline analysis:
• Algorithm 1 : monitoring of spectral lines in ITF output channels
• Algorithm 2 : search of glitches in ITF output channels
• Algorithm 3 : monitoring of the stationarity and gaussianity of the ITF output.
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
27
Online analysis tools
Gai library
Disk Shared Memory
Ethernet
GAI processAlgorithm
Disk Shared Memory
Ethernet
frames
frames
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
28
Online analysis tools
Disk
Shared Mem
Ethernet
Algorithm2
Disk
Shared Mem
Ethernet Algorithm5
Algorithm4
Algorithm3
Algorithm1
: data under frame format
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
29
Online Analysis: current scheme
Full FrameStorage (disks)
Main Frame BuilderOnline ProcessingFrame Distributor
Algo1 datastorage
Algorithm 2 Algorithm 3 Algorithm 1
Algo2 datastorage
Algo3 datastorage
frames
raw data frames
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
30
Online Analysis: futur scheme
Full FrameStorage (disks)
Main Frame Builder Trigger manager
Algorithm 2
Algorithm 3
Algorithm 1 frames
Processed datastorage
frames
raw data frames
6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP
31
Conclusion
Virgo DAQ and online monitoring tools like dataDisplay or Vega+Web have been extensively used since year 2001.
DAQ has shown to be:
• modular (lego pieces with standard connections between them)
• reliable and quite easy to use (and to restart)
• flexible and evolutive
• latency minimized
Beyond DAQ:
• Useful data streams (raw data, trend data, 50Hz data, processed data, …) are under definition
• Online analysis has started