fermilab control system (“acnet”) j. patrick february 17, 2005

Fermilab Control System(“ACNET”)

J. PatrickFebruary 17, 2005

SMTF Review 2/17/05 J. Patrick 2

Fermilab Control System

Overview Applications Central Services Front-ends/Instrumentation Timing Security and Remote Access Summary


Fermilab Control System aka “ACNET” Unified control system for the entire complex

400 MeV Linac 8 GeV Booster Synchrotron 120 GeV Main Injector Synchrotron 1 TeV Tevatron Synchrotron antiproton source – target/debuncher/accumulator antiproton “Recycler” storage ring fixed target lines

Simultaneous operation of Tevatron proton-antiproton collider (storage ring) Antiproton production and storage (~0.5 Hz) 120 GeV fixed target to Meson lab (~0.1 Hz) 120 GeV fixed target to NUMI/MINOS (~0.5 Hz) 8 GeV fixed target to MiniBoone (~5 Hz)


Fermilab Accelerator Complex


Control System Overview

ConsoleApplications

CentralServices

IRMFront-Ends

JavaApplications

WebApplications

MOOCFront-Ends

LabviewFront-Ends

Open AccessClients

Field Hardware

ethernet

Database

CAMAC, VME, PMC, IP, Multibus, CIA, GPIB, …

field bus: VME, SLD, Arcnet, ethernet, …

Servlets Consolidators

Application

Central

Front-End


History Originally developed for Tevatron in early 80’s Substantial evolution over the years CAMAC field hardware -> very broad diversity PDP-11/Mac-16 front-ends

-> x86 Multibus-> VME based processors + Labview

DEC PCL network-> IEEE 802.5 token ring-> ethernet

VAX database -> Sybase VAX central services -> Java PDP-11 applications -> VAXstations

-> Java application infrastructure-> Port of VAX code to Linux in progress


System Scale

200,000 devices/350,000 properties 475 front-ends 90 Sun Netras + 12 Linux PCs that run Java based data

loggers, servlets, open access clients + VAXes to run VAX central services

70 VAXes that run high console level applications + 70 Linux PCs to to parallel VAXes until VAX retirement. X-displayed out to desktop PCs


Device Model <= 8 character devices names

e.g. T:BEAM Informal conventions provide some rationality but no enforcement More verbose hierarchy layer created, but not utilized/maintained

Each device has one or more properties: Reading, setting, analog & digital alarms, digital status & control

Device descriptions stored in Sybase database Scaling transformations defined in device database Add new devices with “DABBEL” command language on VAX

Templates available for various device types View/modify device definitions with GUI Monotype data, arrays are transparently supported Structured data is often done, though support is not transparent

Application (library) layer must properly interpret the data from the front-end and perform required data translations


Device Database GUI


Communication

Applications/Central Services/Front-ends communicate via “ACNET” protocol

Custom protocol now exclusively based on UDP Extensive network/switches etc… Asynchronous request/reply; request multiple reply Timeout/retry mitigates UDP unreliability. “RETDAT/SETDAT” protocol for readings/settings

Can have multiple devices/packet

Fast Time Plot (FTP) returns blocks of readings taken at up to 1440 Hz every 0.5 seconds

Snapshot protocol returns blocks of 2048 points taken at an arbitrarily high rate


Communication Application communication with front-ends passes

through central layer Datapool Manager on VMS via shared memory Data Acquisition/Server Engine (DAE) in Java (via RMI) Consolidation of requests to avoid overloading front-ends Centralizes security, monitoring, error logging

Issues include: Limited packet size and internal device address space. Timestamps not part of protocol packet, applied on data

receiptGPS sync clock event sent to all front-ends, can be included in

structured devices New get/set32 protocol addresses the above, only on centrals

now Lack of transparent support for structured data.

Bridging to network devices that don’t directly support ACNET is via front-ends or Open Access Clients


Database

Central Sybase database Three parts:

Device and node database Application database

Used by applications as they wish Save/Restore/Shot Data database

MySQL used for data loggers, and for some other specialized purposes. Not for general application use


Application Environment Two major high level application environments Console environment

VAX based C language (+ legacy FORTRAN) Port to Linux in progress

Expect some operational utilization in the fall

Java environment Platform independent Windows/Sun/Linux/Mac all supported and utilized Full access to entire control system

Both environments have a standard application framework that provides a common look and feel

All applications launched from “Index Pages” Currently separate pages for Console, Java environments

Also some browser applications of varying styles


Index Page (Console)


Application Index (Java)

Web Browser

Standalone Application


Application Overview Utility applications

Parameter Page Fast Time Plot Data Logger viewer Database Viewer Synoptic displays Diagnostics Save/Restore Alarms …

Machine Specific Applications ~600 Primary + 125 Secondary applications in console

environment >100 applications in Java environment

Mostly utility, shot data analysis


Parameter Page (Console)


Parameter Page (Java)


Fast Time Plot (Java)


Synoptic Display (Console) Graphical Builder

“Lex Draw”


Synoptic Builder (Java)


Synoptic Display (Web)


Sequencer (Console) Automaticall

y Sequence Collider Shots, other activities

Relies on ACL scripting language


Finite State Machine (Java) Graphical Builder Primary current usage is

Pelletron control Keeps track of HV sparks and

beam related trips


Sequenced Data Acquisition Automated acquisition of data for collider shots Configure readings for various stages of collider

shot Snapshots/FTP as well as single scalar readings SDAEdit tool makes it easy to add new things

Data acquired automatically and stored in Sybase database

Summary html reports generated when store goes in Step efficiencies, error reports, luminosity

“Supertable” captures ~200 most popular items Html, Excel, Java Analysis Studio formats

automatically generated Suite of tools to view detailed data Data can be directly read by Excel, Java

Analysis Studio, potentially other external tools Simple Java API for user written code


Alarms Display


Scripting

Accelerator Control Language (ACL) Console environment Access to much of the functionality in the control system VMS DCL style syntax Extensively used in Sequencer

Jython Java API in principle accessible to Jython Some but limited testing/use to date


External Tools

Excel Excel based parameter page Easy cut/paste of Java tables into Excel Excel data can be imported into a variety of tools

MathCAD, OptimA used locally

MATLAB MATLAB supposedly has a standard Java interface, in theory it

should be straightforward to integrate with the control system However we haven’t tried it An attempt to do this with Mathematica > 1 year ago was not

successful, it does not use a standard Java distribution

Java Analysis Studio Currently data logger and shot data plug-ins


Excel Parameter Page


Java Analysis Studio


Save/Restore

Devices may be assigned to save/restore lists

Console applications to save/restore to/from database on user request

Scheduled saves of the entire control system 4 times/day


Electronic Log Based on Oak Ridge

product Web browser client

Perl/cgi back-end Shift log + machine /

subsystem logs Used by A0 API to add entries

programmatically, including images Both Console and Java

environments Java application

framework includes standard menu item to make posts.


Central Layer

Open Access Clients (“virtual front-ends”) Java Servlets

Bridge requests from outside firewall

Request Consolidation Alarms server Front-end download Other things …


Open Access Clients Processes with no user interface, always running

Obey same communication protocol as front-ends Compare to EPICS “Soft IOC”

Several classes: Utility – Data loggers, scheduled data acquisition, virtual devices Calculations – Both database driven and custom Process control – Autotune of fixed target lines Bridges –to ethernet connected scopes, instrumentation, control

systems… Easy access to devices on multiple front-ends Friendlier programming environment than VxWorks front-ends

Framework transparently handles ACNET communication Access to database, other high level operating system features Do not provide hard real-time response

Clock events via ethernet multicast > 100; Almost all run in Java framework; VAX framework deprecated


Data Loggers ~70 instances running in parallel on central nodes

~80 GB compressed storage per node in MySQL database Circular buffer; wraparound time depends on device count/rates

Data at maximum 1 Hz rate extracted to “Backup” logger every 24 hours Backup logger is available online => data accessible forever

Generally allocated to various departments to configure as they wish No central management of archiving lists, so inevitable duplication

Tasks follow the Open Access Client architecture Simple GUI to specify logging lists

Single readings or snapshots; on clock events or at fixed rates to 15 Hz

Programmatic logging interface available for custom loggers Variety of retrieval tools available

Standard console applications, Console and Java environments Web applications Java Analysis Studio programmatic APIs


Front-Ends

Interface field hardware to the high level control system

Several architectures MOOC – Minimally Object-Oriented Communication

Very wide diversity of functionality Processors all VME (VXI) based

Very wide diversity of field hardware supportedLinks to many other form factors such as CAMAC, etc.MVME 162 (68k) and 2400 (Power PC)VxWorks Operating System

~275 in the system


Front-Ends - IRM IRM – Internet Rack Monitor MVME 162; PSOS operating

system Chassis includes 64 channel x

1 KHz 16 bit continuous digitizer 8 x 10-20 MHz option available

8 bytes external digital I/O 8 x 12 bit DAC outputs Expandable via IndustryPak I/O ~125 in the system Eight used at A0 photo-injector

Variety of functions A few at TTF for modulators


Front-Ends

Linac front-ends VME based Power PC processors Non-MOOC infrastructure; tied to 15 Hz Linac cycle rate ~50 in the system

HRM – Hotlink Rack Monitor Evolution of IRM architecture I/O chassis separate from processor chassis 64 x 10 KHz ADC; up to 2 per chassis 10-20 MHz snapshot board Timestamps included with ADC data 8 byte digital I/O per chassis Multiple chassis’ per PPC processor VxWorks (not MOOC) Initial build of 15 units in progress


Labview Front-Ends General control/acquistion

from high level controlsystem

Interact via ACNETprotocol written inLabview VI language No FTP/Snap

Non-local consoleaccess viaPCAnywhere,Timbuktu etc.

~25 in system Instrumentation:

Flying Wires Sample Bunch Display …

Both PCs & Macs


Field Hardware

Wide variety of hardware supported CAMAC, VME, VXI, Multibus, CIA, … Too extensive to describe here…

Support for wide variety of commercial instrumentation Scopes, spectrum analyzers, signal generators, DVMs, … Ethernet->GPIB; direct ethernet Through front-ends; through OACs; direct in some cases Mostly in console environment, ports to Java as needed

Siemens-Moore APACS PLC supervisor Used for cryogenic control Interface through Open Access Client using library from

company Other PLCs also supported through front-ends


Bridge to External Systems

Accelerator NETwork control system (AccelNET) is a software package designed for control of electrostatic particle accelerator systems.

Runs on PC hardware under the Linux operating system

Modular Linux programs that communicate to each other through a central database server using TCP/IP.

CAMAC- GROUP3 I/O ACNETAccelNET Bridge


Timing and Control Networks TCLK (Clock Events)

10 MHz Manchester encoded signal with 8 + 2 bits/event 256 possible events defined for the complex Decoding hardware/signal generators for variety of buses Standalone version of this system used at A0 photo-injector

MDAT (Machine Data) 10 Mbit/sec serial link; 8 bit type + 16 bits data per frame Originally developed to rapidly distribute ramp currents around the

main ring and Tevatron; usage has broadened Beam Sync – revolution markers derived from RF States

State devices defined Any node may issue transition request, gets multicast everywhere Can be distributed via MDAT

Beam permit/abort CAMAC interfaces….


Security

Control system network inside a firewall Restricted access from/to outside the accelerator division

network

Privilege classes assigned based on account, service, (node) Control from where applications may be run Control which programs/users/nodes may do settings

Emphasis on accountability as well as security Setting log Application log


Remote Access

Console environment via remote X display Administrative controls on usage, particularly settings etc.

Java applications can be launched from anywhere from a browser via Web Start Access to inner parts of control system only via VPN Applications to be run remotely w/o VPN must be designed for

thatApplications that work via servlets;“Secure Controls Framework” – Subset of data access API

implemented in this way, better performance than servlets/xml

Access from outside the standard application environments xml-rpc is preferred method Readings of any device; setting of “virtual” devices No setting of hardware associated devices Used by all current experiments


ACNET Summary

Should meet requirements for SMTF Has been/is being continually modernized

Extensive experience at FNAL. Less time to get a system running A broad range of people could contribute quickly

Limited experience outside of Fermilab Remotely accessible but full system not portable Documentation…

fermilab control system (“acnet”) j. patrick february 17, 2005

Documents

hz120 gev fixed target

protocol packet

hz8 gev fixed target

secondssnapshot protocol

retdatsetdat protocol

ends x86 multibus vme

acnet protocolcustom

vax central services70