1.7 beam control: methodology, practical experience and ...ssrf.sinap.ac.cn/ocpaschools06/school...

=== Accelerator Control System ===Accelerator Control System ===

Basic ConceptsBasic ConceptsCase StudyCase Study

Compiled by Hsu, Kuo-Tung (NSRRC)

1.7 Beam control: methodology, practical experience and system hardwareJuly 30, 2006

Basic Course

2

Features of the Accelerator Control SystemFeatures of the Accelerator Control System

• There are many kinds of accelerator（Ring、Linac、Pulse・CW、Collider、Light source ring、Electron・Heavy Ion…) dependent upon its application。

• Geography Scale form table-top to 102 km, all equipments are distributed around the facility.(KEKB Ring ccircumference 3 km, Injector 600 m)

• Control points from 102 (small facility) to 106 (ILC)。(TLS: ~ < 50 K, SSRF ~ >300 K, DLS ~ 450 K, KEKB ： > 105 k, LHC > ILC > 106 ~ 107 K)

3

Linac

制御室

KEKBring

KEKB Accelerator KEKB Accelerator Complex (~ 3 km)Complex (~ 3 km)

SLAC Accelerator SLAC Accelerator Complex (~ 3 km)Complex (~ 3 km)

CERN Accelerator CERN Accelerator Complex (27 km)Complex (27 km) ILC Accelerator Complex ILC Accelerator Complex

(30 km + 30 km)(30 km + 30 km)BEPC (220 m x 2)BEPC (220 m x 2)

HLS (66.1308 m)HLS (66.1308 m) TLS (120 m)TLS (120 m)

SSRF (432 m)SSRF (432 m) SpringSpring--8 (1432 m)8 (1432 m)

Spallation Neutron Source ~ 100 - 1000 mMedical accelerator ~ < 100 m

4

Technology for Technology for AcceleratorAccelerator• Optics - Machine Parameters -• Magnet (Dipoles, quadrupole, sextupole, Corrector/trim,、

Solenoid,…..)• RF (Conventional vs. Superconducting ）

=> Large current, stable beam acceleration and storage• Beam Monitor

=> High performance beam monitors are essential for Precision beam, control- Beam position, beam intensity, profile, …..

• UHV technology=>Beam lifetime、high current => heat load

• Injector• Insertion Devices (Conventional, novel, in-vacuum, resolver,

polarization switching, superconducting, … ）•• Control SystemControl System=> Tight all subsystem together, components together, Nervous

system, High efficiency, reliability, operation

5

What is accelerator control systemWhat is accelerator control system？？Computer Control SystemComputer Control System

• Computers（UNIX Computer、PC、Local Controller）System for accelelerator subsystem control（Magnet, power supply, vacuum, RF…)

• Interconnected by computer network• High reliability storage (RAID, NAS…)

Timing control system• Timing signal (trigger) generation, distribution• => Beam Timing、rf Timing、rf Monitor、Beam Monitor

Timing、Kicker/Septum Timing, experimental timing

Safety System（Interlock）• Machine protection、Human protection

6

（（Large ScaleLarge Scale））Features of Accelerator ControlFeatures of Accelerator Control

• Reused of the existed equipment – especially on the upgrade of existed machine。（Pertain past Investment）

KEKB used many TRISTAN 。（Tunnel、Magnets・Part of power supply、control system

equipments）=> Reduce $$$

• Long Operation time、Reliable operation is essential。(KEKB ring: 6000 hr/year、 Injector:7000 hr/year)

= regular maintenance： 8h/2 week, Long term maintain：2 month (Summer）+end of year =

• Long life time、renew some equipments usually。

7

Computer ControlComputer Control・・Operation Requirement (1)Operation Requirement (1)

• Present clear picture to operator, machine physicist, and subsystem experts

=> GUI, status panele.g. Vacuum system condition、 RF powerー、 current of

magnets……・ Notice operator for specify events=> Alarm・ Accelerator system on-line and off-line analysis=> Archive System：Data storage=> Falut reason tracking=> Machine parameters variation tracking（trend）。 (RF

phase drift）, …etc.=> Correlation between machine parameters（Improve

beam Quality）

8

Compare between measured and designed machine parameters、Provide an environment for accelerator model simulation environment

=> AT, SAD, software environment…etcBeam control

=> Various feedback loop to stable the beame.g. Orbit feedback、energy feedback…etc.

Highly functionality in beam control：Bucket selection, bunch-by-bunch feedback, filling pattern feedback, ……etc…Save, restore accelerator parameters setting

Computer ControlComputer Control・・Operation Requirement (2) Operation Requirement (2)

9

• All support group・Physics group share the information of the machine on-line

（Lateral communication）=> Web brower for real-time operation log=> Daily meetinge.g. Trip report、Tuning report、Ideas exchange

- KEKB Electronic Log Book -Injector: AccessZope (Python script)

Computer ControlComputer Control・・Operation Requirement (3) Operation Requirement (3)

- NSRRC e-log -

10

Accelerator Control System Accelerator Control System –– Important Issues Important Issues

• High AvailabilityHigh Availability=> High system availability（Low fault rate、fast recovery form fault）=> Redundancy are essential…• ScalabilityScalability=> Performance proportional to number of installation、fast response• FlexibilityFlexibility=> 一部機器の置き換えに対応Solid installation methodology, many installation。⇒Standard module available。

TRADEOFFS BETWEEN ISOLATION, SECURITY, ROBUSTNESS, AND TRANSPARENCY

11

Basic Concepts:

Change in computer hardware and software is rapid on the scale of accelerator construction schedules.

Software development must begin early to be ready in time.Most of the investment in a control system goes into the software and front-end electronics.For front-end components the use of industrial standards is now quite safe as systems such as VME

have shown a long lifetime.Software development should be based on the new industrial methods.Object oriented tools have proven to be powerful for control system design.Implementation on more than one platform leads to better design and helps to prepare for the

inevitable evolution of computer systems during the project lifetime.

Users

Users of an accelerator control system: the operations expertsthe machine engineersthe accelerator scientists the administration of the facility.

Each of these groups places different demands on a control system, each (except possibly the administrators) putting an equivalent load on the overall system. The operator needs a reliable and simple system every minute, but the demand at any time is not very high. The engineer and the scientist may need a lot of information, but only at very specific times, depending on the nature of the situation.

Control System Hardware Architecture

12

Technology Development Technology Development -- ICICVacuum Tube (1904: Diode）=> Generate heat、Large power consumption、large

dimension…

• 2000: CPU (4 x 107 transistors ）

• 1952: Transistor

• 1958: IC (102 ~ 103 transistors）(FET transistor)

• 1971: Microprocessor (~ 2300 transistors）

Intel 4004

• 2006: Core 2 Duo CPU (65 nm, > 108transistors ）

http://www.tgdaily.com/2006/06/06/intel_p065_announced/

13

Moore's law is the empirical observation that the complexity of integrated circuits, with respect to minimum component cost, doubles every 24 months. It is attributed to Gordon E. Moore, a co-founder of Intel.

Growth of transistor counts for Intel processors (dots) and Moore's Law (upper line=18 months; lower line=24 months)

Electronics, an American trade journal published until 1995, was best known for publishing the April 19, 1965article by Intel co-founder Gordo Moore in which he outlined what came to be known as Moore's Law.

Moore's law

http://en.wikipedia.org/wiki/Transistor_counthttp://en.wikipedia.org/wiki/Intel

14

Technology Development Technology Development -- ComputerComputer

1945: ENIAC (Electronics Numerical Integrator Computer, PennevaniaUniv.)

•Military purpose（ballistic calculation）

•Vacuum tube 18000 pcs、weight 30 t 、150kW…

•Manual wiring programming

•No reliable enough…(availability 50% ～ 90%)

1958: Born of integrated circuit

1964: IBM360General purpose compute（business, scinetific）IC based machine

15

MinicomputerMinicomputer

1965: PDP-71970: PDP-111978: VAX11/780 VMS1980: MicroVax

VaxStation, AlphaStation, ..VAX 11

PC, Work stationPC, Work station

1976: Apple I1977: Apple II1981: IBM-PC(16-bit)…

IBM-PC/XTIBM-PC/ATIBM-PX/386IBM-PC/486IBM-PC/PentiumIBM-PC/Pentium-III…Core 2 Duo

Apple II

Technology Development Technology Development –– Computer (cont.)Computer (cont.)

IBM 5150

Apple I

http://en.wikipedia.org/wiki/Image:IBM_PC_5150.jpghttp://en.wikipedia.org/wiki/Image:Apple_I.jpg

16

* Use proof standards- Vendor independence (flexibility)- Increased life cycle (portability)

* Use modular and distributed hard/software- Incremental growth (flexibility)- Nearer to equipment (less wiring)- Increased fault tolerance- Working load sharing- Geometrical demands ... etc.- Computer hardware should be easy to change in order to take

advantage of the latest technology; this would apply to the console workstations, front end crates and networks.Applications software should be capable of running under different operating systems. This would be possible if there existed a well-defined application program interface (API) for a control system. Decoupling this API from network and hardware details is essential.

As Simple As Possible, As Complex as Necessary!

Basic Guidelines of Control System for Accelerator Applications

17

* Use reliable industry products- Increased MTBF- Power fault tolerance- Environmental limitations (heat dissipation, op. temp., etc...)- Local field service

* Commercial products, both hardware and software, should be easy to incorporate into the control system,* Experience teaches us lessons !

- Buy it ! don’t built it !- Hardware is cheaper than software- Plan early- Develop prototype !- Use existing database- Use object oriented method !- Application software is written easier than system software !

Assure a good development environment !

Basic Guidelines of Control System for Accelerator Applications – Cont.

“State-of-the-Art” can mean “cutting edge”, but it can also mean “what actually works now”.

18

In summarizing the progress of the control system architecture at the International Conference on Accelerator and Large Experimental Physics Control Systems 91 (ICALEPCS’91) B. Kuiper asserted that the system architecture issue was resolved and presented a “Standard model”.

Standard ModelThe “Standard model” consists of a local area network (Ethernet,

FDDI, ATM, ...etc.) providing communication between front-end microcomputers, connected to the accelerator, and workstations, providing the operator interface and computational support.

Non-Standard Model - Extensions to the standard model

Control System Architecture:The Standard and Non-Standard Models

19

Control system Standard ModelControl system Standard Model

Field Network

Accelerator Equipment

Other Networks

Data Base Operator Interface

High Speed Network

Application Development

VME/ VXICAMACPLCPC/ C-PCI

RFVacuum Magnet

Local Controller

etc...SafetyAcc. Structure

High-LevelBeam Control

• =Network Distributed System==> All connected devices are distributed in a widespread area

• = Multi-layer=- Devices Layer、Upper Layer: Operator、Middle Layer divided-=> Middle layer: Different devices, added devices, and absorbed change。=> Ensure ””Scalability, flexibilityScalability, flexibility””

20

Standard Standard LayerLayer StructureStructure


Local Control Layer

Middle level servers(Host Computers)

Control application Layer(Host/ Client Computers)

= Device Layer =• All component、subsystem are controlled by Local controller 。• Connected by high speed network (Ethernet) to upper layer

= Middle Layer =•Host Computer=> UNIX WS, Linux PC, Windows Server…

= Upper Layer（User Side） =•Operator Interface (OPI)=> X-Window System, VNC (Virtual Network Computing)Remote computer display technology。

21

Merit of multiMerit of multi--layer control system layer control system (1)(1)Orbit correctionOrbit correction

= Upper layer computer（Operator side） =• GUI interface• Orbit correction program（correction algorithm）=> Correctors Steering strength list

= Middle layer =• Mapping corrector name to correspond power supply controller • strength to excitation current conversion• Security control (Allow, Deny?)⇒Distributed set command to various local controller

Timing and synchronization requirement ?

= Device layer（Accelerator hardware side） =• Local controller of PS set current• Set success？ (Read back ?) => NG : some specify measure should take


Local Control Layer



22

Merit of multiMerit of multi--layer control system layer control system (2)(2)Without middle layerWithout middle layer

Lower layer（machine side）：• Varity of local controller can supported form many different kind devices。= Middle layer =• none= Upper layer（operator side） =• Operation program（applications）need knoe the information of controlled devices in lower layer => Mag.・controller location=> Security control=> Need knoe the different of various devices

e.g. : Orbit correction、Optics、Quad-BPM… have different hardware and configuration

=> Expandability is not good=> Expandability is not good

Mag. PS Controller

Local Control Layer


VME PLC CAMAC

?????

23

Merit of multiMerit of multi--layer control system layer control system (3)(3)With middle layerWith middle layer

= Lower layer （machine side） =Varity of local controller can supported form

many different kind devices。= Middle layer =• Hind the different of lower layer=> Security management=> Interpret the command from upper layer=> Machine update（change・increase）、only minor charge on middle layer。= Upper layer（operator side） =• Operation program （applications）is independent form local controller。=> Mag. current setting is independent form local controller (User program）• Local controller is hinded by middle layer。

(*) Multi(*) Multi--layer software layer software hierarchyhierarchy

Mag. PS Controller

Local Control Layer



VME PLC CAMAC

!!!

24

Front-end computerBus

Instruments busCAMAC, VXI, PXI, IEEE-488, Ethernet, LXI … etc.Computer back-panel busISA bus, PCI bus, VME/VME64/VME64x/VME64xP bus, VXI bus, PXI, CompactPCI (cPCI), ... etc

Filed level controllerEmbedded controller

Bitbus node processorPLCVME (IOC)Ethernet based devices

WebDAQ (Web-based data acquisition)FieldPoint (National Instruments)Smartlink (Keithley)LXI…

25

Hardware - Devices level computerVME crate system/cPCI crate system

CPU moduleDigital input/output module, Analog input/output moduleIntelligent controller (Network, IEEE-488, Motion, ... etc.)Special module (Delay generator, Amplifier module, ... etc.)Operating systemControl software package

PCI/O modules

Software - Devices level computerVME crates system

Real-time operating systempSOS+VxWorksLynxOS (real-time UNIX)RT-LinuxRTEMS

Setting processDynamic database upload processI/O processClosed-loop control loopSystem service process

PCLinux, Windows NT, Special control process

26

== Devices Interface ==== Devices Interface ==-- Local Controller (Field bus) Local Controller (Field bus) --

•• Interface with accelerator devices directlyInterface with accelerator devices directly•• RealReal--time vs. non realtime vs. non real--time (speed consideration)time (speed consideration)

Beam monitor => Fast measurementvacuum pressure、… => Medium speedRoom temperature、cooling water temperature…=>

slow。•• Which kind bus is proper for your applications?Which kind bus is proper for your applications?=> Dependent purpose => Dependent purpose 。。(High speed(High speed？？Slow speedSlow speed？）？）(*) High availability is essential。

• General and standard bus (VME, PLC, CAMAC…)=> Adopt standard industrial products as possible。=> Vendor support(*) Only develop special module which are not commercial available

27

--CAMAC CAMAC ––Computer Automated Measured And Control, IEEE583Computer Automated Measured And Control, IEEE583

• Technology of 1960’• Still play an important role in some facility

(KEKB keep many TRISTAN hardware)

Crate controller + Network(@Injector)

Serial highway（@Ring）、上connect to upper layer VME

Add Intelligent (Linux based controller）

Aging, Cooling, Power supplyCard edge connector no good

28

-- VME and its offspring VME and its offspring ––VersaVersa Module Europe, IEEE1014Module Europe, IEEE1014

• VME32 (1982), (1) data transfer bus, (2) priority interrupt bus, (3) arbitration bus, and (4) utility bus. ’90 high reliability, popular in accelerator

control filed。High reliable DIN connector （stable mechanical design and stability）Support various CPU and OS=> PPC, Intel, RTOS, Linux, Windows…

•VME64 (1995)support for 64-bit transfers.

•VME64 Extensions, VME64X (1998)5-row DIN P0 connector geographical addressing voltages pins for 3.3V maintenance bus, and EMI, ESD front panel handles and keying per IEEE 1101.10.

29

VME2eSST (1999)dual edgesource synchronous data transfer (2eSST)320Mbytes/sec.

VITA 31.1 Gigabit Ethernet on VME64x adds GigE to backplanes via a P0 connector

VITA 41, VMEbus Switched Serial , VXSVPX (VITA 46)

New 7-row high speed connector rated up to 6.25 Gbps

Choice of high speed serial fabrics PMC and XMC (VITA 42) mezzanines Hybrid backplanes to accommodate VME64, VXS

and VPX boards

Slightly expensive

30

VMEbus Software

VMEbus has the largest software base of any computer architecture. That's quite a claim, but it is supported by the fact that there are over 100 known, commercial operating systems running on VMEbus. Other, proprietary operating systems are also known to exist.

UNIX Style OS WINTEL Style Real time OS

SolarisSunOSBerkeleyATTLinux

DOSOS-2Windows 3.1Windows 95/98/NTWindows XPVistas

VxWorkspSOS/pSOS+LynxOSQNXRTLinuxRTEMS (Real Time Executive for Multiprocessor Systems)

31

-- cPCIcPCI / PXI/ PXI ––

Synchrotrn SoleilCPCI system and Patch panels

Function Référence Manufact DescriptionVIDEO PXI-1409 NI 4 channels monochrome image acquisition Multiplexage VIDEO PXI-2593 NI 500 MHz 16 x 1 Multiplexer/Matrix Profibus CP353 KONTRON 1 Profibus LinkRS232 PXI3538/9 ADLINK 8 RS232 linkGPIB PXI-GPIB NI GPIB 488,2 HS intarfaceCounter PXI-6602 NI 8 counters / timers 32 bits, 80 MhzHigh speed DIO CPCI 7300 ADLINK 32 DIO 80 MB, Handshaking

CPCI 7248 ADLINK 48 DIO TTLCPCI 7432 ADLINK 32 DI opto (24V) + 32 DO opto (500mA)

CPCI 6208V ADLINK 8 channels +/- 10V 16 bitsCPCI 6208A ADLINK 8 channels 0-20mA 16 bits

synchrone AO +/- 10 V PXI 2502 ADLINK 8 channels +/- 10V 12 bits 1MHz

PXI 5122 NI 2 channels +/- 10V 14 bits 100MS/sDC211 ACQIRIS 1 channel +/- 10V 8bits 4GS/s

PXI 2010 ADLINK 4 channels +/- 10V 14 bits 2MHzPXI 2005 ADLINK 4 channels +/- 10V 16 bits 500 kHzPXI 2204 ADLINK 64 channels +/- 10V 12 bits 3 MHzPXI 2205 ADLINK 64 channels +/- 10V 16 bits 500 kHz

asynchrone AI+/- 10 V

Standard board selection

basic DIO TTL or isoleted

Fast AI +/- 10V

asynchrone AO+/- 10 V or 0-20 mA

synchrone AI +/- 10 V

CPCI crate customizedCPU Pentium MWindows chosen as Operating System in the cratesFor each standard board, Controls groups provide:

– A TANGO DeviceServer– A patch panel system

32

--PLC PLC ––Programmable Logic ControllerProgrammable Logic Controller• Sequence Control

• Simple I/O module

• Simple Logic

• Safety system、Klystron、

Vacuum、Magnet control

•High reliability

• SIEMENS S7 300 PLC used• Commercial interconnect product

used• Call for subcontractor:

– a PLC specialist helped us to • Analyse the planned

architectures• Prepare the PLC

integration and programmation

– To write the Cabling folder – To integrate the PLC

• Machine and BeamLine groups :• Do the functional

analysis• Write PLC software part • Test the developped

systems

Synchrotrn Soleil

PLC system andPatch panels

33

-- MiscellaneousMiscellaneous Bus Bus ––• VME

=> VXI（Instrumentation）

• PCI

• RS-232

•USB 1.0 / USB 2.0

• IEEE-1394

• PC-104 (Standalone)

……….

34

Server Computer

Servers are computers that provide a service to other processes.Front-end servers : Device serverMiddle layer servers (group servers):

Help client programs to get information from the huge numbers of data acquisition equipment.Collect the front-end data and add proxy services such as “intelligent mirrors”.All devices information of a subsystem is available in the middle layer, high order functions can be implemented.User and program file systems, data bases for all machine parameters, on-line machine optic calculations and simulations, sequencer to automate the machine operations and alarm and error logging.

35

Console computer

Console level computersWorkstationsPCControl serverWeb serverOperating systemControl software packageMan machine interface

Software Environment- Console level computer

Operating systemVMSUNIX (various dialect and Linux)Windows Windows 2000, Windows

XP)Linux

Database managementNetwork accessAPI (application program interface)Control applicationsGUI (graphical users interface)Utilities program

36

ClientsA client is a program that uses services from the servers in the control

system. An application program or a display process are typical examples of clients. These programs run on the operator consoles, in the tunnel during installation and maintenance and in the offices of the machine groups. Other applications run in the hardware and software development labs. All these programs should use the same communication protocol and the same library to talk to the servers. This is important since it should be avoided to rewrite the same software for different environments.Client program is a wide-field and consists of some generic programs:

Platform independentAccess all devices in standard wayTools to display alarms and other device errorsTools to access all data in the system (parameter page)Tools to display historical trends and other plotsStandard programs that allow creating dedicated applicationsGenerate and display synoptic representation of devicesStore and recall machine parametersIntegrate device data into office products like text editors and spread

sheetsAccess device data for analytical program like MATLAB, PV-Wave, …Interface to machine simulation programsProvide control system I/O for industrial tools like LabVIEWDefine automated sequencing of machine operationsCommercial tools will play an increasing role in the client programs

37

A fieldbus is a network that connects sensors, actuators or complex front-end input/output devices to the local device servers.

RS-232C/RS-485/RS-422IEEE-488 (GPIB), LXI (LAN eXtension for Instrument), Bitbus, Mil-Std 1553B, ProfiBus, ARCNETCANIEEE-1394 Fire Wire, USBEthernet

... etc.There are many different system, a control system should to focus on very

few standards to reduce the maintenance costs since modern fieldbuses are quite complex.

Many standard electronics instruments equip with IEEE-488 interface. New generation instruments (PC instruments) built in a PC which equip Ethernet.

Fieldbus have some limitation inDistance, devices number, speed, …

Usually the software effort to communication to the RS232 devices is underestimated.

Networking – Fieldbus

38

Ethernet 10 Mbps/100 Mbps/1000 Mbps/10 GbpsATM OC-3C, OC48, OC-196 …FDDIArcnetToken RingWLAN (wireless local area network, Bluetooth …)

... etc.Wide Area Networking

* ISDN* ATM

TCP/IP is a standard networking protocol.

To keep the network load low and predicable, the network is segmented into several parallel lines to the front-ends. => Real-time performance in some case.

Maintenance and managing consideration also lead a good segmentation of subsystems.

Networking – Local Area Network (Control Network)

39

Operating system

UNIX and Windows NT in server and consolesReal-time system in front-ends.VxWorks

pSOS+LynxOS (real-time UNIX)RT-Linux

………Evolution direction -> object oriented systems.For maintenance reason, the number of different operating systems should be kept small.Design based on more than one system leads to a better portability in the future.Many modern operating system dialects have “soft” real-time capabilities and provide a multi-threaded, multi-tasking environment.LINUX as a free UNIX system that runs surprisingly stable, and is a good choice for low budgets.

Control system – software structure

40

Communication interface

Communication interface is a basis of a control system. It defines the application program interface (API) for the data exchange between clients and servers.API must be based on standard protocols.Network layer -> TCP/IP Internet protocol.Higher level protocols are transported for the basic TCP/IP services.Requirements for the top level areTransport a well-defined set of data objects.Convert the data between different computer architecture.Provides common methods e.g. translations of the data structure.Access to all data of devices in a consistent way (including archived data).Interface to standard desktop applications like spread sheets and to special controls applications.Provide an on-line names service to query existing devices and their properties.Defines a unique way for names of device address and their parameters.Allows access to single and multiple devices in one call.Send parameters ads data to the device in a read call to allow atomic requests.Client programs must get device address resolutions from a name server and not by recompilation.

41

Database

Data bases are used to store the parameters of the machines, theconfigurations of the servers and the archived on-line measurements of devices. ORACLE or other standard commercial products with interfaces to different computer types and a solid market share as indication of continuing support should be selected.

Oracle, Sybase, DB2, MS SQL server, …etc.For on-line archiving and configuration data of front-end servers, a big database is too slow and voluminous to fit into a small front-end. A small real-time database usually adopted for time critical applications.

Central database for configuration management.Static database : store configuration parameters like names, constants, calibration coefficients, attributes, alarm levels, fieldbusaddresses, etc.Historic database: log data over long periods of time, long-term history data are stored into a relational database and retrieved off-line.Real-time database (Snapshot database) :

Store the state of the machine.

42

Example of database in Swiss Light Source control system:

An Oracle database is used to hold all static and dynamic system configuration information. This information includes all devices, their properties, and their characteristics. This information is used to generate EPICS configuration data for each VME crate, information for the CDEV directory service, and also hold information directly used by physics application programs, such as the distance between a magnet and a BPM or the magnetic length of a magnet. Equipment specialists can manipulate the Oracle database via a web interface to update operational parameters (such as operational limits for a power supply).

43

Application programs

Machine physics applicationsMachine operation oriented applicationsDevice oriented applicationsArchive/Data viewerDatabase management

... etc.Man machine interface (MMI)Graphical User InterfaceWeb browser interfaceWireless interfaceWeb technology applied in control systemHTML, CGI, JAVA applet make it easier to access the data on the web regardless the platform you use.Controls developmentMaintenanceAccessibilityOn-line logbookAccess the logbook more readily form office computerSearch capability – find information in the logbook quickly and efficiencyMachine status Logistics, Inventory, User interface (CGI, Java, ... etc.), Network management, Embedded Java in real-time environment

… etc.

44

Object-oriented technology applied in accelerator control

CORBA (Common Object Request Broker Architecture) is a recent standard that provides a mechanism for defining interface between distributed components. Its most distinguished assets are platform independence, in so far as the platform hosts a CORBA Object Request Broker (ORB) implementation and language independence, as ensured through the use of the Interface Definition Language (IDL).Middle-ware, software-bus, real-time databaseChannels, tags, points (EPICS, Vista, Wizcon)Object/action (ESRF, Elettra, LEP)COBRA remote objects (ANKA)

Accelerator modeling

Many computer programs and a variety of models exist for the design of accelerator lattices and the correction of errors.

The operation of a modern accelerator is rely on model drive controls and feedback systems.

Modeling techniques is used to find and correct both linearand nonlinear optics errors during the commissioning of new accelerators, and during the operation of these accelerators.

It is also true when we extend the operation of an existing accelerator to a new regime.

45

Typical beam-based accelerator modeling and controlOrbit correction (SVD)Beam-based alignmentDebugging linacs (Model-Independent Analysis, MIA)Debugging storage ring optics

With betatron oscillationsWith closed orbit shifts

Coupling correctionBetatron oscillationsClosed orbit shifts

Lattice diagnostics

註解: Server服务器(server)，在香港和台灣稱為伺服器是指：•一個管理資源併為用戶提供服務的電腦軟體，通常分為文件伺服器（能使用戶在其它電腦存取文件），資料庫伺服器和應用程序伺服器。•運行以上軟體的電腦。(維基百科，自由的百科全書)

46

註解: Middleware"In a distributed computing system, middleware is defined as the software layer that lies between the operating system and the applications on each side of the system." , http://middleware.objectweb.org/

一、为什么要中间件计算机技术迅速发展。从硬件技术看，CPU速度越来越高，处理能力越来越强；从软件技术看，应用程序的规模不断扩大，特别是Internet及WWW的出现，使计算机的应用范围更为广阔，许多应用程序需在网络环境的异构平台上运行。这一切都对新一代的软件开发提出了新的需求。在这种分布异构环境中，通常存在多种硬件系统平台(如PC，工作站，小型机等)，在这些硬件平台上又存在各种各样的系统软件(如不同的操作系统、数据库、语言编译器等)，以及多种风格各异的用户界面，这些硬件系统平台还可能采用不同的网络协议和网络体系结构连接。如何把这些系统集成起来并开发新的应用是一个非常现实而困难的问题。

二什么是中间件为解决分布异构问题，人们提出了中间件(middleware)的概念。中间件是位于平台(硬件和操作系统)和应用之间的通用服务，如图1所示，这些服务具有标准的程序接口和协议。针对不同的操作系统和硬件平台，它们可以有符合接口和协议规范的多种实现。

http://en.wikipedia.org/wiki/Operating_systemhttp://middleware.objectweb.org/

47

也许很难给中间件一个严格的定义，但中间件应具有如下的一些特点：满足大量应用的需要运行于多种硬件和OS平台支持分布计算，提供跨网络、硬件和OS平台的透明性的应用或服务的交互支持标准的协议支持标准的接口由于标准接口对于可移植性和标准协议对于互操作性的重要性，中间件已成为许多标准化工作的主要部分。对于应用软件开发，中间件远比操作系统和网络服务更为重要，中间件提供的程序接口定义了一个相对稳定的高层应用环境，不管底层的计算机硬件和系统软件怎样更新换代，只要将中间件升级更新，并保持中间件对外的接口定义不变，应用软件几乎不需任何修改，从而保护了企业在应用软件开发和维护中的重大投资。

http://wiki.donews.com/index.php/中间件

48

CORBA

CORBA(Common Object Request Broker Architecture)通用物件請求代理架構是軟體構建的標準。CORBA標準由物件管理組織（OMG）設立並進行控制，CORBA定議了一系列API，通信協議，和物件/服務信息模型用於使得異質應用程序能夠互相操作，這些應用程序用不同的程式語言編寫，運行在不同的平臺上。CORBA因此為定義明確的物件提供了平臺和位置的透明性，這些物件是分散式計算平臺的基礎。通常來說，CORBA把用其他語言開發的程式碼和關於該程式碼能力和如何調用該程式碼的資訊包到一個套裝(package)中，包成套裝的物件則可以在網路上被其他程式(或CORBA物件)調用。在這個意義上來講，CORBA可以被看作是一個機器可讀的文件檔格式，類似於標頭檔(header)，但是具有相當多的資訊。CORBA使用一種介面定義語言用於刻畫物件將呈現出來的介面。CORBA又規定了從IDL到特定程式語言，如C++或Java，實現的映射。這個映射精確的描述了CORBA資料類型是如何被用戶端和伺服器端實現的。標準映射的有Ada、C、C++、Smalltalk、Java、以及Python。還有一些非標準的映射，為Perl和Tcl的映射由這些語言寫的ORB實現。(維基百科，自由的百科全書)

http://zh.wikipedia.org/w/index.php?title=%E7%89%A9%E4%BB%B6%E7%AE%A1%E7%90%86%E7%B5%84%E7%B9%94&action=edit

49

software bus

A programming interface that allows software modules to transfer data to each other. Although "bus" is traditionally a hardware term for an interconnecting pathway, it is occasionally used in this manner when the focus is on internally transferring large amounts of data from one process to another.

Review — Software Buses

Middleware choices affect architecture and architecture constrains choice of a middlewareMiddleware provides interoperability between disparate applications across heterogeneous platforms

Software Buses- Software buses provide an assembly language for software composition- Bus technologies represent a concrete counterpart to conceptualconnectors- Facilitate distribution, heterogeneity and resolve interaction mismatches- Various research infrastructures - Q, Polylith, Field

50

Subsystem connected to the control systemVacuum System

Vacuum PumpVacuum GaugeInterlock and Protection

Power Supply System (Magnet System)DC Power SupplyAC Power SupplyPulse Magnet Power Supply (Injection & Extraction)Cooling WaterInterlock and Protection

Radio Frequency SystemGunLinacKlystronModulatorCavityTransmitterRF AmplitudeRF Phase Interlock and ProtectionSupport System

51

Beam Instrumentation SystemScreen MonitorsBeam Position MonitorsIntensity Monitors

DCCTCT... etc.

Synchrotron Radiation MonitorScrapers

Timing System* Injection, extraction* Bucket addressing

52

Insertion Devices* Wiggler

*UndulatorGap controlTune correctionOrbit compensationInterlock and protection

* Elliptical Undulator=> Gap control

Phase controlTune correctionOrbit compensationInterlock and protection

* Superconducting Wiggler/Undulator=> Power supply control

53

Tune correctionOrbit compensationInterlock and protectionPhoton Beam line and experimental station

Utility, Environment, Radiation SafetyVarious control related subsystem (some examples)

Digital global / local feedback systemEPBM systemTune measurement systemProfile measurement systemLattice function measurement systemInstrumentation automationTune feedback systemTransverse coupled-bunched feedback system

Gain, delayLongitudinal coupled-bunched feedback system

Gain, delay, control rules, …... etc.

Control System – Case Study in Hardware, Software and Networking

54

-- RedundancyRedundancy、、Distributed loadDistributed load --

• Computer redundancy ( File server)

=> Cluster, Fail Over

• Network redundancy

=> Main line, Backup line

• Storage redundancy

=> RAID、high reliable Disk system

• Power redundancy

=> UPS (Uninterrupted Power Supply)

=> How swapable power module, two power modules

• Cache server

=> Read data caching、reduce CPU and Network load。

RAID Disk

LAN

Host-A Host-B

Client Machine

Fail Over

Trouble!!

55

User Facility:

One of the primary goals is to maximize availability of beams for the experimenters.

High I/O densityHot swapPlug and playEMC protectionLoosely coupled

56

== Summary ==== Summary ==== Accelerator control system (from small to hughmachine） ==

Control system standarization – Standard model ---•Networked distributed control system•Multi-layer hierarchy structure（Hardware・Software）hidden in lower layer====== High High Availability, Scalability, FlexibilityAvailability, Scalability, Flexibility======

•– Redundancy and reliability• Local controller(Fieldbus)

= VME, CAMAC, PLC, VXI,……

• Information sharing mechanism=> Web-based– Operation and Malfunction Log=> Better communication system, good supports -

57

== EPICS Toolkits ==== EPICS Toolkits ==• EPICS (Experimental Physics and Industrial Control System)• A control system toolkits• EPICS Collaboration (’89: Los Alamos, ANL）。More than 100 facility

adopted• Short time, high performance control system development is possible• HLS, BEPC II、SSRF, ……

Project Control Points (Record Number) IOC No. Host No. Operation Condition

APS 300,000 170

4

4

4

4

6

8

22

22

1000

---

~ 250

~ 300

20 Operation

HERA Cryo 6,000 1 Operation

TTF Injector 600 2 Operation

Keck II Telescope 1,500 6 Operation

PSR 2,500 6 Operation

Duke FEL 2,500 3 Operation

PEP-II RF 8,400 2 Operation

BESSY II 15,000 10 Operation

APS Beamline 15,000 10 Operation

KEKB 300,000 5 Operation

KEK Injector 70,000 --- Operation

DLS 450,000 ~ 20 In commissioning

SSRF > 300,000 ~ 20 In construction

EPICS

58

EPICS

From RTEMSWikiEPICS (http://www.aps.anl.gov/epics/index.php) is an acronym for Experimental Physics and Industrial Control System. It is a set of software tools and applications which provide a software infrastructure for use in building distributed control systems to operate devices such as Particle Accelerators, Large Physics Experiment Detectors, Lasers, and major Telescopes. Such distributed control systems typically comprise tens or even hundreds of computers, networked together to allow communication between them and to provide control and feedback of the various parts of the device from a central control room, or even remotely over the internet. EPICS has been used as the foundation for a variety of custom physics and scientific control systems. EPICS is a world-wide collaboration. It is open source. Retrieved from "http://www.rtems.com/wiki/index.php/EPICS"

EPICS

http://www.aps.anl.gov/epics/index.phphttp://www.rtems.com/wiki/index.php/EPICS

59

Introduction: What is EPICS?

• A collaboration of the controls groups of many research organizations that use the EPICS tool-kit.

• A distributed architecture that supports a wide range of solutions from small test stands to large integrated facilities.

• A set of tools that reduces software application and maintenancecosts by providing:

Configuration tools in place of programmingA large installed base of tested softwareA modular design that supports incremental upgradesWell defined interfaces for extensions at every level

EPICS

60

The EPICS Collaboration

• Over 100 independent projects in North America, Europe and Asia

• Applications in particle physics, astronomy, and industrial control

• Distribute software over the network• Independent development, co-development and

incremental development of code done by members• Problem reporting and resolution via e-mail exploders• Documentation available on WWW sites• Large collaboration meetings to report new work,

discuss future directions, explore new applications, and explore new requirements for existing codes

• Small design groups from multiple labs meet to discuss design issues on significant codes: Channel Access, CDEV, Archiving and MMI

EPICS

61

Distributed Architecture

• EPICS is physically a flat architecture of front-end controllers and operator workstations that communicate via TCP/IP and UDP

System scales through the addition of new computers

Physical hierarchy is made through bridges, routers, or a gateway

Network bandwidth is the primary limiting factor• EPICS software architecture is client/server based - with

independent data stores providing read/write access directly between any two points

Local name services mean automatic integration of new components

Point-to-point communication supports automation

EPICS

62

Distributed Hardware Configuration

Workstations:SunHPDEC/AlphaSilicon GraphicsPC

I/O Controllers:VME, VXI, PCI, ISA

Field I/ORemote I/O and SignalConditioning:CAN-Bus, Industry PackVME, VXI, PCI, ISACAMAC, GPIBProfibus, Bitbus, SerialAllen-Bradley, Modbus

Field I/O Field I/O Field I/O

EPICS

63

Standard Operating System Software Configurations

Workstations OS:SunOS, SolarisHPUXDEC-UNIXSGIXWindows NTLinux

I/O Controllers OS:vxWorks

Field I/O Field I/O Field I/O Field I/O

EPICS

64

Distributed Software Architecture

ca-server

process DB

device sup

ca-clientCDEV

VMSVxWorksUNIXLinuxWindowsNT

UNIX

ca-client CORBA ACE

VxWorksWindowsNTSolarisLinuxRTEMS

ca-server VxWorks only

EPICS

65

Many tools are available in the EPICS tool-kit

• EPICS tools are connected via the Channel Access client/server libraries

• Server Interfaces:Process DatabaseGateway (CA-Client - GDD Library - Portable

server on Solaris)• Client Interfaces

Process Database LinksSequencer (programmed with State Notation

Language)Data Visualization PackagesData Analysis PackagesModeling and Automation Packages

EPICS

66

Channel Access Client/Server Libraries

Sequencer

Channel Access Client

LAN/WAN

Operator Interface


Database Links


Channel Access Server

EPICS Process Database

Client: Provides read/write connections to any subsystem on the network with a channel access server

Server: Provides read/write connections to information in this node to any client onthe network through channel access client calls

Services: Dynamic Channel Location, Get, Put, MonitorAccess Control, Connection Monitoring, Automatic ReconnectConversion to client types, Composite Data Structures

Platforms: UNIX, vxWorks, VMS (Client only), Windows NT

TCP/IP & UDP

EPICS

67

Channel Access

Sequencer


LAN/WAN

Operator Interface


Database Links


Channel Access Server


Performance:68040 over 10 Mbit EthernetGets

Propagation Delay 2 mSThroughput 7.8K /sec

PutsPropagation Delay 1 mSThroughput 17K /sec

MonitorsPropagation Delay Dependent Throughput 10K / sec(Typically 10% channels have monitors)(memory use in IOC - 2 Meg / 60 connections)(30% network load for 10K monitors / second)

Increase bandwidth with Routers, Bridges, Higher speed networks and EPICS gateway

TCP/IP & UDP

EPICS

68

EPICS Process Database Provides Data Acquisition and Control

ca-server

process DB

dev support

ca-client

Record types are the basic elements: AI, AO, BI, BO, Motor, CALC, PID, SUB etc.…Records consist of fields for: SCHEDULE, I/O, CONVERT, ALARM, MONITORFields hold runtime values: VALUE, TIMESTAMP, ALARM CONDITION, etc.…New record types are easily addedConfigured using CAPFAST, GDCT, JDCT, Relational DB, Text Editor at the workstationLoaded as ASCII records into vxWorks at boot timeMost fields can be read/written through the channel access client interface during operation

LAN A Channel Access server provides connection, get, put, and monitor services to this databaseA Channel Access client provides access to process DBs in other IOCs

EPICS

69


ca-server

process DB

dev support

ca-client

Record execution time varies from record type to record typeAI on a 68060 is ~12,000/second (50% idle)AI on a 68040 is ~6,000/second (50% idle)

Fastest periodic scan rate is dependent on vxWorks clock tick (typically 60 Hz)Interrupt scanning is limited by the CPU bandwidth (interrupt delay ~33usec)Name resolution - 10,000/second - runs at the lowest priority2,500 records use around 1 Megabyte of memoryAssignment to particular physical I/O is distinct from process logic - ASCII device type

LAN

EPICS

70

Channel Access Client Codes Provide a Variety of Functionality

Visualization: EDD/DM, MEDM, Dataviews, SL-GMS,tcl/TK, Visual Basic, Visual C++, Labview, IDL, SAMMIJAVA

Alarm Handler: ALH State Programming: SNLOther Clients: PV-Wave, Mathmatica, Matlab, Probe, Orbit Lock,

Correlation Plots, Striptool, X-Orbit, SAD, CNLS-netSDDS, ARTEMIS, Archivers, Save/Restore

Operator InterfaceChannel Access ClientLAN

EPICS

71

EDD/DMAn Interactive Display Editor and High

Performance Display Manager

EDD is an interactive display editor for creating operator screensDM is a display manager to activate screens created using EDDConnections are made to channels - typically record-nameDM monitors channels and only updates the screen when there are changesMonitors from channel access are cached - not queuedASCII display formats can be read/written for version upgrades/text editing


The Channel Access Client provides name resolution,monitors and puts to any data stores with a CA server

EPICS

72

EDD/DM An Interactive Display Editor and High

Performance Display Manager

Static objects include: text, line, box, ovalMonitor objects include: text, bars, meters, indicatorsControl objects include: text, slider, menu, buttonsPlot objects include: strip chart, Cartesian plots, smith chartsRelated display call-up includes parameter passingColor Rules are available for all objects independent of dynamic channelVisibility modifiers are available for static objects DM does 2,000 updates per second on a SPARC IPC2 second display call-up for 100 dynamic and 1000 static elements


The Channel Access Client provides name resolution,monitors and puts to any data stores with a CA server

EPICS

73

The Alarm HandlerAlarm Viewing, Logging and Annunciation

Alarms are detected in the data stores - most commonly the process databaseAlarms are configured into an alarm hierarchyLogging, annunciation, acknowledgment and collection is configured for each groupAll lower levels of the hierarchy inherit the disable status from aboveOperator logging tracks alarm disable, alarm acknowledge and silence commands500 Alarm condition changes per second can be handled on a SPARC IPC


The Channel Access Client provides name resolution,monitors alarm changes, writes group alarm status

EPICS

74

State Notation LanguageA State Machine Implementation in EPICS


The Channel Access Client provides name resolution,monitors and writes to state programs

EPICS

SNL is preprocessed to be converted to C codeConstructs are provided for easy channel access interfaceConstruct are provided to define state programs, state sets and transitionsSNL implements the Mealy Model - actions are only taken on transitionSNL runs in the vxWorks environmentEvents are supported for synchronizing state sets within a state programSNL does not have a server The process database is used for variables to be exported

75

EPICS Collaboration : Cooperative Development of a Scaleable, Flexible Tool-kit

• The fundamental performance and functionality is scaleable and easily configured

• Clean interfaces for clients, new record types, data stores and hardware promote independent development, support ease of reintegration, and protect against obsolescence

• Cooperative collaboration gives members laboratories a larger pool of talent to support their controls

• Continual improvements allow members to expand functionality, performance, reliability and function while taking advantage of latest technology

EPICS

76

Record Type Descriptionaai Analog data array devices inputaao Analog data array devices outputai Analog data devices inputao Analog data devices outputbi Binary data devices inputbo Binary data devices output

calc 計算式評価calcout 計算式評価及び値出力camac General purpose CAMAC record

compress Data compressdfanout Data fanout

eg イベントジェネレータ recordegevent イベントジェネレータのイベント生成

er イベントレシーバ recorderevent イベントレシーバのイベントビットパターンevent Generate software eventfanout プロセスリンクの fanoutgpib General purpose GPIB recordgsub 多入力サブルーチン record

-- EPICS standard record EPICS standard record --EPICS

77

-- Database file *.db Database file *.db --EPICS

78

-- Support Hardware Support Hardware -- EPICS

–– Vendor List Vendor List --

79

-- IOC Process IOC Process --

CA server

DB Access

Runtime Database

CA TCP CA UDP Task/ Chan

DB lib/ Record support

Device support

Driver support

SequencerScanner

Event

Hardware Interface

EPICS

80

- MEDM (Motif Based Editor & Display Manager) -

その他に、

•EDD/DM (EPICS OPT Display Editor/ Display Manager) - 最初のDM -

• jdm (java based display manager) – javaのDM -

EPICS

81

Tango is the new generation of the ESRF control systemIt is an evolution of the old ESRF TACO control systemIt is developed in close collaboration between Soleil and ESRF

TANGO is a CORBA based control system being developed by the ESRF and Soleil synchrotrons in France.

TANGO uses CORBA for doing network communication.CORBA is a language independant standard for implementing distributed objects on the network. CORBA is an evolving standard and is continuously adding new features which are of interest for controls e.g. real-time, embedded, component model.

http://www.esrf.fr/tango - the TANGO website

TANGO

82

What is TANGO ?

• It uses CORBA as middleware between applications and the hardware

• It is object oriented• It supports object oriented language (Java/C++/Python)

and three OS (Linux, Solaris and Windows)• It is interfaced to commercial software used in our

institutes (Igor, Matlab and LabView)

• ESRF, Soleil, ELETTRA, ALBA, ….

83

TANGO PHILOSOPHY

name - every device has a name which is unique in every instance of TANGOproperties - device are configured by their properties. Properties are persitantlystored items which describe certain aspects of device configuration e.g. channel address, baudrate, min, max etc.attributes - each device can have a set of data attributes. Attributes have a fixed description which defines minimum and maximum values, alarm and warning levels, event trigger levels etc. Attributes can be events too. commands - each device can execute a set of commands, commands accept one input parameter and one output parameter. A parameter can be an array.events - are asynchronous attributes which are sent to all subscribers.server - devices are instantiated in a process called a device server. This is a container for a collection of devices. It provides an administration interface torestart the server, read out the black box, reinitialise devices etc.polling - device commands and attributes can be triggered automatically. This is doing by polling devices inside servers. The polling thread is used to trigger events, fill the data cache, trigger actions.multi-threading - device servers are multi-threaded.The omniORB implementation of CORBA used by TANGO is thread-safe and allows writing threaded applications using TANGO.

84

TANGO SERVICES

database - a system wide database using MySQL is provided for persistantstorage of device properties and device names.

naming - a naming service is provided via the database which allows devices to be found independant of which host they are running on. Clients and servers from multiple instances of TANGO can connect with each other.

event - TANGO uses the omniNotify implementation of the CORBA Notification service.

logging - a logging service which uses log4j is provided for all devices.archiving - archiving of historical data is provided.

groups - groups of devices can be constructed on the fly and controlled as a single device.

api - a high-level application programmer’s interface is provided in C++ and Java. The api provides stateless connection management.

starter - TANGO servers can be automatically started by a starter daemon.

85

TANGO TOOLS

pogo - a graphical tool for generating device classes in C++ or Java. Removes a large part of the tedious work involved and makes device class development very rapid.

logviewer - a graphical tool for viewing and filtering log messages.

jive - a graphical tool for viewing and modifying the database and testing devices.

devicetree - a generic graphical tool for testing devices and building simple monitor panels.

atk - a graphical application toolkit with a large number of graphical viewers for various data types. Can be run standalone or be used to build new graphical applications.

astor - a graphical tool for supervising a TANGO control system e.g. starting device servers, checking device servers are running etc. and testing devices.

86

TANGO BINDINGS

C++JavaPythonMatlabLabviewIgor

87

An Accelerator Control Middle Layer Using MATLAB

Gregory J. Portmann, et al.

The Middle Layer software provides a library of functions that access either the machine hardware via EPICS (MCA, LabCA, or SCAIII) or the AT simulator.

88

Computer Room Ethernet Switch1000/100/10 Mbps

ControlServers

TLS - Control System Hardware Architecture

Injector

1.5 GeVStorage Ring

SRRCAccelerator

System

Simatic 135PLC

Storage Ring Ethernet Switch1000/100/10 Mbps

Injector Ethernet Switch100/10 Mbps

ControlConsoles

(Workstation/Unix)(PC/Linux)

Control Room Ethernet Switch1000/100/10 Mbps

MVME 147/pSOS+ (8 nodes)PPC/LynxOS (22 nodes)

PC/Windows

SRRCIntranet

ProgramDevelopmentLaboratory

VME

HOST

I/O

I/O

I/O

I/O

I/O

I/O

VME Crate

VME

HOST

I/O

I/O

I/O

I/O

I/O

I/O

VME Crate

VME

HOST

I/O

I/O

I/O

I/O

I/O

I/O

VME Crate

VME

HOST

I/O

I/O

I/O

I/O

I/O

I/O

VME Crate

VME

HOST

I/O

I/O

I/O

I/O

I/O

I/O

VME Crate

VME

HOST

I/O

I/O

I/O

I/O

I/O

I/O

VME Crate

VME

HOST

I/O

I/O

I/O

I/O

I/O

I/O

VME Crate

VME

HOST

I/O

I/O

I/O

I/O

I/O

I/O

VME Crate

To Ethernet

GPIB/ENET 100Controller

Electronics Instruments

DiskArray

89

TLS Control System Software Structure

Command Processing Program

Child Task forSpecial Services

Static Database Dynamic Database

Device Control Service ProgramStatic Database Service Program Data UpdateService Program

DatabaseLoggingProgram

Real-TimeTrend Display

LabVIEWInterface

Matlab MEXInterface

Data Acquisition Program

Child Task for Setting Devices

Control Ethernet

BasicSoftware

Componentson ILCs

BasicSoftware

Componentson Consoles

DevicesLevel

(M68K/pSOS+)(PPC/LynxOS)

(G4 PPC/VxWorks)

ConsolesLevel

(Compaq WS/Unix)(PC/Linux)

Injector

1.5 GeVStorage Ring

SRRCAccelerator

System

Self-ConfigurationTable

Dynamic Data Pool

Shared Memory

MachineParameters

ServiceRoutines

Machine Modeling(MPAP,APAP)

Control Applications

Data Access Routines

Data Access Routines

November 2001

Matlab ScriptsAP

MATLAB

LabVIEWVI

LabVIEW

System StatusDisplay

Alarm CheckingProgramParameter PageProgramData Logging

& ArchivingHardware DiagnosticsProgram

AP

90

Bird eye view of SCSS

8GeV Electron Accelerator

SCSS PrototypeAccelerator

8GeV Electron Accelerator

SCSS PrototypeAccelerator

500m

91

GigaEther Switch

Office LANProgramDevelopment

DMZ LAN

OperatorConsol

Gun

Web

Magnet & Interlock

DB & File server

FirewallFirewall

1000BASE-SX

Buncher S-Band Bunch Compressor C-Band 1 C-Band 2 ID

H-PCF

Magnet Power Supply

PLC

VME

Control System of SCSS

92

Software

• Message And Database Oriented Control Architecture (MADOCA)– Designed by SPring-8 control group in 1994– MADOCA is scalable, covering wide range of

accelerators and beamlines.– Develop the prototype accelerator control

software to adapt the full-scale SCSS– Using X-Mate, a GUI builder enables rapid

prototyping and easy development of GUIs


93

Message server

Collector

Equipmentmanager

devices

Collector &Poller

VMEs

System V IPCONC/RPC

Console

Sybase

SQL data access

Loggingdatabase

Parameterdatabase

Alarmdatabase

Databaseserver

Alarm

Accessserver

Accessserver

Accessserver

RF, Mag …

devices

User FunctionUser Function

GUIGUI GUI


94

• Operation consoles are IA-32 workstations running RedHat Enterprise Linux 3.

• Database and file servers are shared with the SPring-8 control system.– We use Sybase ASE12.0 for the RDBMS– MC/ServiceGuard is used for High Availability

Cluster.• VME controllers use IA-32 processor boards (Pentium

III 700MHz & Pentium M 1.1GHz) running Solaris 9 x86 with CF cards.

• Use Programmable Logic Controllers (PLC) and the Remote I/O (OPT-VME) – Reduce the number of I/O signal wiring

• Gigabit Ethernet is used for a backbone of network– VLAN & PoE technology

Computing system

95

Developed hardware components

• High-speed A/D and D/A VME boards.– Both are 238MHz clock and 12bit

resolution– Fully digital controlled RF low-level with

IQ modulator and demodulator is needed to provide a stable beam for FEL.

– A/D boards are used for rf-BPMs and CTs.

DA

FPGA

DA

I

Q

Modulator

AD

FPGA

AD

I

Q

Dem

odulator

ee--

PhaseFlip

I

Q

• Master trigger and a trigger delay VME board.

CKD Q

CKD Q

24bitscounter

238MHz

5712MHz

Trigger

FPGA

– Fast GaAs logic devices to synchronize the 5712 MHz– FPGA (238MHz clock) with 24 bits counters– The time jitter is 0.7ps.

~ Comparable to bunch length = 0.4psec

96

--- Injector –--•In-house developed RPC (Remote Procedure Call)•Control program development: C/C++•GUI: Tcl/Tk（one set Python、SAD) script language・Database (homemade): ASCII text base.•Cache serverto reduce CPUand Network loads•Main server: Tru64/UNIX(alpha)•=> Move to Linux Cluster is next step•Web server, file server

KEK (Injector) Control SystemKEK (Injector) Control System

Number of Equipment in Number of Equipment in KEKB InjectorKEKB Injector

佐藤政則, et al.

97

MS-Windows Interface for Operation X-Window Interface for Commission

Touch Panel for Operation EPICS Gateway to KEKB-Ring

Main Network (FDDI, Ethernet)

Main Computer System (Unix)

Equipment Level Network (Optical Ethernet)

VME

Beam Monitor

Magnet

Trigger

PLC

Klystron

Magnet

Vacuum

CAMAC

Trigger

VXI

RF Monitor

PC/GPIB/

RS232CGUN

Beam Monitor

others

MS-Windows Interface for Operation X-Window Interface for Commission

Touch Panel for Operation EPICS Gateway to KEKB-Ring

Main Network (FDDI, Ethernet)

Main Computer System (Unix)

Equipment Level Network (Optical Ethernet)

VME

Beam Monitor

Magnet

Trigger

PLC

Klystron

Magnet

Vacuum

CAMAC

Trigger

VXI

RF Monitor

PC/GPIB/

RS232CGUN

Beam Monitor

others

• OPI: MS-windows, touch

panel、X-Window（Linux/PC)。

• Host computer：Tru64(hp)

Operation：3 sets、 Developmet:

2 sets

• Local controller：

VME,PLC,CAMAC….

KEK (Injector) Control SystemKEK (Injector) Control System

98

KEKB (Ring) Control SystemKEKB (Ring) Control System

--- Ring ---•EPICS Toolkit•IOC(Input output Controller)として大量のVMEを使用•SAD, Pythonスクリプト言語の採用•Relational Database (ORACLE)の使用

=> 1970年:IBM社1件のデータは、複数の項目(フィールド)の集合。データの集合をテーブル（表）で表現。・ID番号や名前などのキー・データを利用し、データの結合や抽出を容易に行なう。・データベースの操作にはSQL(Structured Query Language)と呼ばれる言語を使うのが一般的。

http://e-words.jp/w/IBM.htmlhttp://e-words.jp/w/IBM.html

99

Control Network

Laboratory Network

abco1 abco2

acsad

PC

KEKB, PF-AR Operation

Operator Console

EPICS Software DevelopmentRelational Database (ORACLE)

KEKB OperationBeam Optics SimulationCommissioning&Study tools

102 IOCs

(VME Computers)

VXI Main FrameMXIbus

CAMAC Serial HighwayCAMAC

GPIB GPIB Device

LAN-GPIB

ARCNETMAGNET P.S.

RS-232CMeasuring Instruments

Terminal Server

•• EPICS base systemEPICS base system• OPI: X-Window（MAC)• Host computer：Tru64(hp), hp-ux• Local controller：VME x 100、multi-kinds of LocalBus=> GPIB, VXI, CAMAC•VME : EPICS IOC- VxWorks (RTOS) -


100

Touch Panel Touch Panel

Virtual TP GUI

• Injector main OPI

• Touch panel dispaly + Knob +PC-98

=> X-Window base TP


101

Swiss Light Source (SLS)

Data Storage – Several TB RAID arrayPhysics Applications - A dedicated modelingServer, parameters. The model server is accessed by a CorbaOperator Interface Level - PC/LinuxNetwork level - Fast ethernetEquipment Interface Level - ~ 180 VME64xTiming - Event systemDatabase- Oracle database is used to hold all static anddynamic system configuration information.Diagnostics - DBPM, …Linac and RF - TurnkeyMagnet Power Supplies - 500 PSVacuum - Siemens S7 PLC,AlignmentInsertion devicesSLS beam lines S. Hunt, et al.

102

Diamond Light Source (DLS)

Control System Architecture

• Based on EPICS using Two Layer Model– Primary interface to CS through VME IOCs– Use VME64x, IP carriers, IP Modules and transition board

for rear connection – Hot Swap capability

• Will use PLCs to manage interlocks for protection– Avoids Watch Dogs on IOCs and allow warm reboot of IOCs – Omron CJ PLCs for low end applications, eg Vac Valve control– Siemens S7/300 and S7/400 for high end applications, eg

Linac and Cryoplant• Serial Interface to Instrumentation

– Potentially several thousand– Serial support through Stream Device and ORNL Serial

M.T. Heron, et al.

103

Hardware

• Development– Linux for Development– Running EPICS R3.13.9 on the machine and R3.14.7

on beamlines, but ultimately moving to R3.14.8.2 – Using Tornado 2.2

• Consoles – PCs running Linux RH 9.0– Will support Win2000/XP

• IOCs– VME64x – PPC604 Processor boards, evaluating MVME5500– Will use IP carrier and modules– Primarily 7 slot crates– Currently 257 IOCs with 450000 PVs (Machine

Only)

104

Synchrotron Soleil

Hardware Architecture

Profibus

Supervision / Control • SCADA Applications• General Services: o Archivingo Configuration

• TANGO

EquipmentsEquipments

ETHERNET

Compact PCI System

Profibus

PLCs

Motion System

A. Buteau, L.S. Nadolski, et al.

105

User applications User’s development environments

Java applications : configuration,

monitoring, logging, system administration

Tango java ATK

« Ready to use » high level applications

ArchivingService

SCADA

Software ArchitectureThe TANGO « device oriented » philosophy

Hardware access devicesDevice Device

TANGO Devices

Device

Equipment and subsystem devices

Process & calculation devices

Device

TANGO Software bus

Synchrotron Soleil

106

Shanghai Synchrotron Radiation Facility (SSRF) - Linac

G.B. Shen, et al.

109

BEPC II

110

BEPC II (cont.)

111

UserInterface

Connectivity

IOCServices

GUI(tcl/tk)

CA

GUI(edm)

CA

GUI(Java)JCA

CA

Epics Records

SNL Device/driver

HardwarePSC/PSI, I/O

CA CA

SUB

www

OracleDatabase

QUERY

Oracle HW

Config info

VDCTText editor

GenerateDb

Template

IOC db*.db file

Macrosubstitusions+

dbloadImport/Export

data

GUI builder(edm,tcl/tk,java)

Database(VDC,text editor, Oracle JERI)

Standard applications

(Archiver/AHLStriptool,vxStats/Save&restore)

GUI builder(edm,tcl/tk,java)

Database(VDC,text editor, Oracle JERI)

Standard applications

(Archiver/AHLStriptool,vxStats/Save&restore)

C.H. Wang, et al.

BEPC II – Software Architecture (cont.)

112

BEPC II (cont.)

The console consists of 14 unit 14 21-inch LCDs will be installed in the console 8 32-inch LCDs and 5 42-inch PDPs will be on

the display wall in next month.

The console consists of 14 unit 14 21-inch LCDs will be installed in the console 8 32-inch LCDs and 5 42-inch PDPs will be on

the display wall in next month.

C.H. Wang, et al.

113

BES III

BESIII数据获取系统示意图

表4.12-1 BESIII探测器数据量估计

探测器子系统信道数 VME机箱读出数据量

(Mbyte/s)机群处理数据量

(MByte/s)记带数据量

(Mbyte/s)

MDC（T+Q） 13600 46.6 28 21

EMC 6272 24.8 17 13

TOF+CCT 896 2.3 1.6 1.2

MUC 9088 2.4 1.6 1.2

触发 400 6.4 6.4 4.8

小计 30256 82.5 54.6 41.2

PPC/VxWorks

114

HLS

115

Presented by Guobao ShenKEKB Control Group

HLS (cont.)

116

PETRA III Control SystemThe control system uses a multi-layer architecture linked by the integrating middleware or software bus TINE (Threefold Integrated Network Environment), a set of communication protocols and services developed over the past years as the core of the HERA 1/2 control system

117

LHC

Control System Architecture

118

As shown in Figure in previous page, the LHC control system has three hierarchical layers of equipment communicating through the CERN Technical Network, a flat Gigabit Ethernet network using the TCP-IP protocol.

Starting from the bottom of Figure:

• At the equipment level, the various actuators, sensors and measurement devices are interfaced to the control system through three different types of front-end computers :

VME computers dealing with high performance acquisitions and real-time processing; these employ a large variety of I/O modules. Typically, the LHC beam instrumentation and the LHC beam interlock systems use VME front-ends.

PC based gateways interfacing systems where a large quantity of identical equipment is controlled through fieldbuses, such as the LHC power converters and the LHC Quench Protection System.

Programmable Logic Controllers (PLCs) driving various sorts of industrial actuators and sensors for systems such as the LHC Cryogenics systems or the LHC vacuum system.

LHC (cont.)

119

• At the heart of the control system, powerful UNIX servers host the operational files and run the LHC applications:

Application servers hosting the software required to operate the LHC beams and running the Supervisory Control and Data Acquisition (SCADA) systems.

Data servers containing the LHC layout and the controls configuration as well as all the machine settings needed to operate the machine or to diagnose machine behaviour.

Central timing which provides the cycling information of the whole complex of machines involved in the production of the LHC beam and the timestamp reference.

Processes running in these servers will communicate with the equipment access machines using various mechanisms and protocols such as the Common Object Request Broker Architecture (CORBA) and TCP-Modbus.

• At the control room level, consoles running the Graphical User Interfaces (GUI) will allow machine operators to control and optimise the LHC beams and to supervise the state of key industrial systems. Dedicated fixed displays will also provide real-time summaries of key machine parameters

LHC (cont.)

120

BESSY is a contractor to the PTB for building and operating the MLS

Double bend achromat (4-fold symmetry) with two straight sections (RF and undulator)

Ring is ramped from 100 MeV to end energy (200-600 MeV)

The Willy-Wien-Laboratory

The Metrology Light Source

121

Restrictions on manpower enforced a low-effort solution:• EPICS-based (what a surprise…)• ~ 10 IOCs: MVME2100 (controls) & MVME5500 (BPM

system) running VxWorks 5.4.2 (moving to RTEMS?)• ~ 90 PS via CAN bus and i386 controlled AD/DA combo card

(24bit set point resolution)• BPM system (~ 4 mv5500 IOCs) copying the BESSY II

technology• ~ 3 PLCs: RF systems for microtron and ring (external

vendors)• ~ 1 microIOC: vacuum gauges• ~ 4 soft IOCs on Linux: GPIB through GPIB-Ethernet boxes• ~ 3 console machines: PC hardware running Linux (Debian)• Microtron: Danfysik

The Metrology Light Source Control System

122

Control Room

SLS, Switzerland

ESRF (10 year ago), FRANCE

SPring-8, Japan

DLS, UK

123

Control Room (cont.)

SNS U.S.A.Central Control Room in CLO

JLab, U.S.A.

124

deconstructioText: KandicP

n: control roome Carter, JLab

hoto: Greg Adams, JLab Volume 02, Issue 3, April 05, Symmetry MagazineKeeping an Eye on OperationsThe Crew Chief oversees the entire room's operations while the Program Deputy, responsible for the accelerator program, has an alternative view of the action. The Safety System Operator is focused on camera shots of accelerator access points, but the Personnel Safety System (PSS) status is visible to anyone in the room. Accelerator operators credit the new control room design with helping them run CEBAF more efficiently.


125Digital Tigers Zenview™ multi-screen displays


Display Technology

126

“Global Accelerator Network” (GAN). The name refers both to a network of collaborating institutes, and to thecommunication networks that would be required to allowremote participation by distant laboratories in the commissioning and possibly operation of the facility itself. The technology for remote monitoring already exists, and remote access to modern control systems is generally implemented, subject to the security restrictions of individual institutions. One extreme idea was to havethree complete control rooms, making extensive use of video-conferencing technology, one or more each in Europe, Asia and America, with machine control rotatingaround the world from continent to continent. Day shiftsonly!Deep Space Network is an international

“Global Accelerator Network” (GAN)

network of radio antennas that supports interplanetary spacecraftmissions, and radio and radar astronomy observations for the exploration of the solar system and the universe. The network also supports selected Earth-orbiting missions. http://en.wikipedia.org/wiki/Deep_Space_Network

http://en.wiktionary.org/wiki/Networkhttp://en.wikipedia.org/wiki/Radio_antennahttp://en.wikipedia.org/wiki/Interplanetaryhttp://en.wikipedia.org/wiki/Spacecrafthttp://en.wikipedia.org/wiki/Radio_astronomyhttp://en.wikipedia.org/wiki/Radar_astronomyhttp://en.wikipedia.org/wiki/Solar_systemhttp://en.wikipedia.org/wiki/Universehttp://en.wikipedia.org/wiki/Earthhttp://deepspace.jpl.nasa.gov/dsn/gallery/images/canberra4.jpghttp://deepspace.jpl.nasa.gov/dsn/gallery/images/goldstone3.jpghttp://deepspace.jpl.nasa.gov/dsn/gallery/images/madrid4.jpg

127

Timing System

Tasks of a Timing System• Synchronization of accelerator components over the

complex• Steering the beam from source to the “target”: triggering

the beam handling components at right times• Triggering of diagnostics devices and measurements• Scheduling of the accelerator operation

Scope and domains• Tasks for timing systems range from

– synchronization to a single bunch level (“fast timing”)– coarse level synchronization like magnet ramps and

software scheduling (“slow timing”)• commonly but not necessarily different approach for

fast and slow timing– operational sequences (injection control, machine time

allocation and optimization)

128

Fast Timing

• Synchronization at the level of individual bunches or RF cycles (sometimes means also high repetition rate)

• Triggering of a particle source (e.g. electron gun), kicker magnets

• Triggering of diagnostic devices (BPM, current monitors, streak cameras etc.)

• High precision

Slow timing

• Synchronization typically at the level of a machine cycle or a storage ring revolution (~ μsec)

• Fast timing supervision• Ramping of magnets etc. during acceleration• Scheduling of software actions• Control of operational sequences

129

Generation & distribution of timing references

• With any kind of timing, the reference signals (fiducials) have to be generated and distributed to device locations

• High precision fiducials: downconversion from RF, synchronization of multiple sources (RF & AC)

• Low precision: related to high precision fiducials, calculated by programs, spreadsheets, diagrams (operation model like ramp table, fill schedule)

• Distribution method & media (various solutions)

Technology Decisions• Performance requirements are machine dependent

– Stability, precision, resolution• Distribution method and media

– performance, price, complexity• Operational requirements

– how complex patterns of operation have to be executed

130

Direct (baseband) distribution

• Generated signals (and RF) are directly transmitted as pulses

• Synchronization with RF is done at the receiving end, with a delay module (counter)

• High precision, high cost method (cabling)Example system (KEKB fast timing)

• Signal distribution (RF & fiducial) with optical fiber, high performance transmitters & receivers

• Phase stabilized optical fiber used to improve thermal stability (1.3 ps/C/km)

• A fast counter module (TD4V) is used to generate delays and to resynchronize with RF (selected for SLS fast timing, too)

• rapid bucket selection by adjusting the delays (presented in this conference)

• Used only for the most timing-critical applications

131

State code transmission

• Actions (triggers) are encoded in a n-bit status word and transmitted serially

• State word bits correspond to action requests (triggers) to devices

• Status word stream is decoded with receiver modules• Receiver modules deliver the signals to equipment

Example system (DAΦNE)

• A VME DSP card takes as input the timing fiducials (50 Hz) and commands from the control system

• a 31-bit status word is assembled and broadcast by the DSP every 20 μs, using 1 MHz serial link– the status word contains bucket number and

enable/disable bits for timed devices• Receiver card (DSP) receives the status words and

outputs the status word to the local bus

132

Event systems

• Modulated signal transmission• Timing fiducials are transmitted encoded and broadcast to a

number of receivers• Variants:

– use beam synchronous clock as carrier & encode events on the carrier (Fermilab)

– asynchronous transmission (APS, RHIC, KEK)

Modulated transmission

• Globally distributed event stream

RECEIVER

Encoded event stream &beam synchronous clock (carrier)

Outputsignals

Softwareinterrupts

timestamp register

133

Example: the APS event system

• Event generator– hardware and software event inputs– each input corresponds to an event code (8 bits) that is

transmitted on an optical fiber link– RAM for storing and transmitting event sequences– Uses TAXI chip transmitters and receivers, 100 ns

resolution between events

triggerinputs

time

eventreceivers

134

• Event receivers– decode the incoming events and generate output: hardware pulses or

VME interrupt– have fixed width, programmable delay & width and latch (set/reset)

outputs– hardware support for timestamps– programmable response to received events

Example: the APS event system (cont.)

APS Event system upgrade for SLS• Due to its flexibility and good integration in EPICS, we chose to use the APS

system• The manufacturing had to be organized and we decided to use the chance to

upgrade:– use gigabit transceivers for higher resolution (16 bit frames at 50 MHz,

20 ns resolution)– more recent FPGA chips, configurations in flash ROM (card may be

adapted for different applications)– prescalers for different delay resolutions (up to 20 ns)– include a “distributed bus” for synchronous transmission of a number of

signals

135

SLS Event system upgrade for DLS

Structure of the DLS Timing system

136

SSRF Timing System (Event system)

D.K. Liu, et al.

137

BEPC II Timing System (Event system)

Ring Linac

E4434B, 499.8MHzEVG-200

Timing stationRing control room

e+ RF station

e- RF station

spectrometer

transmission line

E4428C In linaccontrol room

250m PLL transmission line

To beam instrumentation station

e gunOM3 optic fiber

EVR-200, Gun-TXin room 102

trigger

e- kicker

e+ kicker

OM3 multimode optic fiber

OM3 optic fiberEVRs to trig modulators EVRs to trig modulators

North

West

Ge Lei, et al.

138

Timing System for Next Generation Accelerators(Optical Frequency Comb Techniques)

J. Byrd, et al.

139

Operations and timing• The timing system is involved in many (all?) accelerator

operations – beam injection– diagnostic measurements– acceleration (ramps)– “synchronous” beam bumps– top-up injection– etc.

• How are these supported by timing systems?

Injection• Task: fill to a specific bunch pattern and current and stop

(start other actions)• SLAC uses a scripting language to describe injection cycles• Sequence editor (Daphne) to generate sequences of actions• CERN uses templates to define timing requirements, these are

assembled into timing programs (presented in this conference)

140

Other tasks• Some of the scripting approaches include all timed tasks• in EPICS, the record structure is used to achieve the desired

functionality (through events and processing links).• Asynchronous operations like beam bumps• Top-up injection; automatic, autonomous operation

Conclusions• Are there chances for standardization of control system

interface, hardware solutions?– Would enhance software sharing, reduce resources needed to

build the basic system and perhaps result in better solutions• Specific applications and needs that cannot be standardized will

always arise• Every accelerator has a different flavour of timing system; is

this really necessary?• The tasks are common, the solutions resemble (but are still

different)– Fast timing: counters, RF distribution, high resolution delays– Slow timing: different flavours of event systems

141

Acknowledgement

Materials in this report are pirated form many literatures.

Thanks of all of these contributed authors.

=== Accelerator Control System ===��Basic Concepts�Case StudyTechnology for Accelerator Redundancy、Distributed load -== Summary == == EPICS Toolkits ==Channel Access- EPICS standard record -- Database file *.db -- Support Hardware -- IOC Process -- MEDM (Motif Based Editor & Display Manager) -

1.7 beam control: methodology, practical experience and ...ssrf.sinap.ac.cn/ocpaschools06/school...

Documents