KEKB Accelerator Control System

KEKB Controls Group

2001-08-29

by T. Katoh

Control System Control Computer System Timing Control System Beam Gate Control System Personnel Safety Control System Communication System Control Room and Operators Consoles

Contents Construction History of KEKB

Accelerator Control System KEKB Control System Design System Hardware Configuration System Software Configuration Central Control Room

Construction History1994.4 Started System Design for KEKB1995.2 Controls Systems Evaluation1995.5 Committee Recommendation to adopt EPICS1995.8 First Version of Computer System Specificati

ons1996.1 Started to Dismantle TRISTAN Main-Ring1996.3 Final Version of Specifications1996.6 Opened the Bids1997.3 abco1 and 15 IOCs were Installed1997.4 Hardware Specifications were Fixed1997.10 BT Lines were Commissioned1998.3 The Rest of IOCs were Installed

Construction History(contd.)1998.4 Started Software Installation with Link-Pers

ons and Engineers from the Company

1998.12 Commissioning of KEKB Accelerators

1999.3 Commissioning with the BELLE Detector

1999.4 Started Designing PF-AR Control System

2000.10 Installation of abco2 and 10 IOCs for PF-AR

2001.2 Started to Dismantle PF-AR Devices

2001.7 Started to Install or Re-install Equipment

2002.1 Commissioning of PF-AR

KEKB Control System Design

System Requirements

Constraints

Basic Concepts

System Architecture

System Requirements Sources

– Equipment Groups• Magnet and Power Supply• RF• Beam Monitor• Vacuum• Beam Transport• Feedback, Facilities, Physics, etc.

– Operations Group– Accelerator Physicists or Commissioning Group– Controls Group

System Requirements(contd.)

Requirements– All the data that are possible to take should be

taken.

– All the data that are taken should be saved for

later analyses.

– All the operation should be recorded for later

inspection.

– All the machine parameters and information about

the machine components should be stored in the

database.

System Requirements(contd.)

Requirements(contd.)

– The Man-Machine Interface should be Operator-

Friendly.

– The Programming Environment should be

Programmer-Friendly.

– The Overall Response Time to an Operator’s

Request should be less than a Second unless

the Progress of the Process is Indicated.

Constraints Use CAMAC as an Equipment Interface

Schedule– Control Systems Final Design January, 1995– Removal of MR equipment January,

1996– Control Systems Installation September, 1996– Completed Hardware Installation March,

1997– KEKB Commissioning December, 1998

Constraints(contd.) Limited Man-Power

– 9 KEKB Controls Group Members• 3 Physicists and 6 Engineers

– 12 Link-Persons from Other Groups• 2 from Magnet and Power Supplies• 2 from RF• 2 from Vacuum• 2 from Beam Transport• 2 from Beam Monitor and Feedback• 1 from Linac, 1 from Physics

Basic Concepts

Standard Model Architecture

International Standards

– CAMAC, VME, VXI, GPIB, FDDI, etc.

Existing Software Environment : EPICS

Separate Computer-bus and Field-buses

“Link-person” System : Applications

“Out-Sourcing” : 5 People from Companies

Standard Model Architecture Presentation Layer (Server)

– Man-Machine Interface(OPI)– Logging, Analyses, Alarm Displays, Datab

ase, Global Feedbacks Equipment Control Layer (IOCs)

– Data Acquisition, Sequence Control, Local Feedbacks

Device Interface Layer (Field-buses)– Hardware Interfaces

The most Important Points Presentation Layer

– Software Development Tools• User/Programmer Friendly Tools

– Ergonomic Consoles• LCDs, Macintosh’s, PCs, PDPs, Flat Desks

Equipment Control Layer– VMEbus : High Reliability, Flexibility

Device Interface Layer– CAMAC : High Reliability, Well-known

Presentation Layer Operator’s Consoles : X-Terminals Database Management : ORACLE 7 Alarm Generation/Recording Data Logging Data Display : CATV Network Simulation : SAD Program High-Speed Network

– FDDI, Distributed Shared-Memory Network, – 100 Base/TX and 10 Base/T

Gateway to KEK Laboratory Network : acsad

Equipment Control Layer Provides Standard Interfaces

– CAMAC Serial Highway Drivers(Hytec)– MXI-bus Drivers for VXI Main Frames(HP)– ARCNET Drivers, GPIB Drivers(NI), etc.

Computers : VMEbus based IOCs– FORCE CPU-40(MC68040)– FORCE CPU-60(MC68060)– FORCE PowerCore6603(PPC 603e)– FORCE PowerCore6750(PPC 750)

Operating System : VxWorks

Device Interface Layer Standard Interfaces

– CAMAC : RF and Vacuum– VXI : Beam Position Monitors– GPIB : RF, Vacuum, Magnet Readouts, et

c.– ARCNET : Magnet Power Supplies

TV Signal Switches– MODBUS+ : PLCs– RS232: Vacuum Measuring Instruments

System Architecture(1) Functional Configuration

Database

Console Console Console

Simulation Logging Alarm Gateway Display

Laboratory Network

Console

CAMAC

VME

CAMAC

VME

CAMAC

CAMAC

Presentatikon Layer

Equipment Control Layer

Device Interface Layer

High-Speed Network

CATV

CAMAC

System Architecture(2) Actual Configuration

Gateway Display

Laboratory Network

CAMAC

VME

CAMAC

VME

CAMAC

CAMAC

Presentation Layer

Equipment Control Layer

Device Interface Layer

CATV

CAMAC

FDDI Network Switch

X-Term X-Term X-Term X-Term

Alarm, Logging, Database, Console, Simulation

Consoles

F/E F/E

System Configuration

Power Supply UnitARCNET Driver Module

GPIB Controller Module

CAMAC Serial Highway Driver Module

CPU Module

Various Equipment

Magnet Power Supplies

FDDI SwitchServer

Workstation

Consoles & Peripherals

Measuring Instruments

VXI Main Frame

RS232 Module

Central Control Building

Local Control Building

VME-MXIbus Driver Module

CAMAC Crates

Central Control Room

Cisco Catalyst 1200 Bridge Cisco Catalyst 1200 Bridge

Phaser 550JX2 QMS2425

Server abco1 ME/RK 460

Cisco C1400 Concentrator

FDDI GIGA Switch

26 Local Control Rooms

Lattis System

810M

X Terminals for Consoles

VME IOC Timing, etc.

abco1 Server Workstation PA-RISC 7200 Architecture

– 120MHz CPU Clock– 4 CPUs– 2GB of Main Memory– 4GB Hard Disk Drives– FDDI Interface– 20GB RAID Disk

acsad Server Workstations Compaq Alpha Server

– 4 Alpha CPUs of 440MHz– 6 Alpha CPUs of 330MHz– 1 GB Memory– 50 GB RAID– FDDI Network Interface– True 64 Unix Operating System

Former KEKB Control System

abco1

acsad

Laboratory Network

KEKB Control Network94 IOCs

PCs

KEKB Accelerator OperationEPICS Software DevelopmentRelational Database

KEKB Accelerator OperationBeam Optics Simulation

Operators’ Consoles

Real-time Control

KEKB/PF-AR Control System

abco2

acsad

Laboratory Network

KEKB Control Network104 IOCs

PCs

KEKB & PF-AR Accelerator OperationEPICS Software Development

Relational Database

KEKB Accelerator OperationBeam Optics Simulation

Operators’ Consoles

Real-time Controlabco1

abco2 Server Workstation PA-RISC 8500 Architecture

– 440MHz CPU Clock– 2 CPUs– 1GB of Main Memory– 36GB Hard Disk Drives– FDDI Interface– 140GB RAID Disk

IOC Configuration

Power Supply Module

ARCNET Driver

GPIB Controller（ NI GPIB 1014）

CAMAC SD（ HYTEC VSD 2992）

CPU Module （ PowerCore 6750 ）

CAMAC Crates

Measuring Instruments

Magnet Power Supplies

FDDI ｰ Ethernet Bridge

Cisco Catalyst 1200

Ethernet HUB

Lattis 810M

Measuring Instruments

VXI Main Frame

RS-232C Module

MXIbus Interface Module

Terminal Server

Cisco 2509/2511

System Monitor Module

X TerminalsRS-232CEthernet Ethernet

From CCR

VME IOC System Subrack

– Schroff 16 Slots Subrack Power Supply Module

– 259 Watts, 5V 35A, 12V 5A, -12V 2A– 14 HP Wide, Schroff MPS8-7746

System Monitor Module– Mitsubishi DRSJ-01

• Remote System Reset• DC Power Lines Monitor

VME Modules Used CPUs

– FORCE CPU-40 MC68040 33MHz– FORCE CPU-60 MC68060 66MHz– FORCE PowerCore 6603e PPC 603e– FORCE PowerCore 6750 PPC-750 266MHz– FORCE PowerCore 6750 PPC-750 400MHz

CAMAC Serial Highway Driver– HYTEC VSD 2992

VME Modules Used (contd.) GPIB Controller

– National Instruments GPIB 1014 ARCNET Driver

– Advanet ARCNET-4 VME-MXI Driver

– Hewlett Packard VME-MXI– National Instruments VME-MXI II

MODBUS+ Interface Controller

Number of IOCs and VME Modules

MG RF MO VA BT MV EN TI PH LI Total

IOC 8 8 20 12 4 20 20 2 1 1 96

CAMAC - 8 - 12 2 - - 2 - 1 25

GPIB 8 - - 12 6 20 20 - - - 66

RS-232C 1 - - 12 - - - - - - 13

ARCNET 96 - - - 4 - - - - - 100

MXI - - 20 - 1 - - - - - 21

Total 105 8 20 36 13 20 20 2 0 1 225

Equipment Connected 2,517 Magnet Power Supplies

– 176 ARCNET Segments 800 Beam Position Monitors

EPICS Records on IOCs 242,597 EPICS Records on 94 IOCs 25,147 EPICS Records on IOCMGD06 2,788.5 EPICS Records in Average

48,149 kB Max. Allocated Memory 1,324 kB Min. Allocated Memory 12,142 kB Ave. Allocated Memory

Hardware Summary Use Standard Buses

– Reliability : VME, Compact PCI, CAMAC• Not ISA, EISA, PCI, Proprietary Buses

– Separate CPU Bus from Field Buses• Isolation, Analog Signal Handling

Choose Components Carefully– Power Supplies– Capacitors, Fans, Connectors

Use Ergonomic Equipment

Control Consoles

TRISTAN Consoles

KEKB and PF-AR Consoles

KEKB Consoles Voice Generator using Power Macintosh Multiple Screen Display for Consoles

– DELL Optiplex NX1– Intel Pentium II 400MHz, 1GHz– Colorgraphic Evolution 4– Macintosh with Multiple Video Cards

X-terminal using IBM Network Station– Power PC 406(133MHz)– 48MB Memory– Disk-less / Fan-less System

Control Consoles 18” TFT LCDs(1280x1024 pixels) 16” TFT LCDs(1280x1024 pixels) 14” TFT LCDs(1024x800 pixels) Book-type Personal Computers

– DELL Optiplex NX1 and IBM Network Station– Multi-Screen Display Controllers– Wireless Keyboard/Mouse

Power Macintosh with 2/4 Video Controllers Six 40” Plasma Displays as TV Monitors Easy to change Configuration

Software

Software Two Language Architecture

– TRISTAN Experiences : NODAL and PCL– Interpretive Language for Applications

• SAD and python• Short Turn-around Time• Safe Debuging Environment• Not Dedicated Application Software Programmer

– Compiler Language for Low-Level Software• C or C++ for Low-Level Software• Quick Response Time• Runs Fast• Realizes Everything You Want

Software (contd.) Relational Database Software

– From the Designing Stage– Put All the Accelerator Information

• Cable Connection Lists• Equipment Parameters

– Physical Parameters : Sizes, Weight, …– Calibration Constants, Fitting Curve Parameters, …

• Histories– Installation Dates, Repair Histories, Costs, ….

• Property Numbers and more …

– Generates EPICS Database Automatically

Software (contd.) EPICS Software Toolkit

– medm• Generated from Relational Database

– Ah : Alarm– Ar : Archiving– SNL : State Notation Language

EPICS Software Used medm SAD python

medm Applications

python Applications

SAD Applications

EPICS Databse Files

Linac Portable CA Server

Linac Console

Linac Main Controls

Linac Sub-Controls

Linac Equipment KEKB Equipment

KEKB IOCs

KEKB CA Clients

Linac CA Server

Linac Portable Channel Access

Application Software in Use SAD 141 medm 74 python 42 Miscellaneous 6 Total 263

EPICS Records on IOCs 242,597 EPICS Records on 94 IOCs 25,147 EPICS Records on IOCMGD06 2,788.5 EPICS Records in Average

48,149 kB Max. Allocated Memory 1,324 kB Min. Allocated Memory 12,142 kB Ave. Allocated Memory

Software Summary EPICS R3.13 Toolkit on HP-UX 10.2 medm for Man-Machine Interface X-Window Server Software on the Terminals python for Application Programming SAD for Accelerator Simulation / Operation VxWorks on IOCs Tornado Cross Software Development Sys. CAPFAST for EPICS Runtime Database Dev. ORACLE 7 for Relational Database Portable Channel Access Server for Linac Controls

VME CPU Board Benchmarks

CPU Board Benchmark Test

FORCE Pcore 6604 FORCE CPU-40

CPU Power PC 604e MC 68040

Clock 200 MHz 25 MHz

Memory 16 MB 16 MB

L2 Cache 512 kB --

CPU Load Arising from Scanning Database

PCore 6604 CPU-40

1.0 sec 0.60 % 5.8 %

0.5 sec 1.2 % 12 %

0.2 sec 3.0 % 27 %

0.1 sec 6.1 % 56 %

Transactions Time Required for Channel Access

PCore 6604 CPU-40

ca_search 953 us 1,189 us

ca_put 23 us 109 us

ca_get 58 us 118 us

ca_put&ca_get 76 us 246 us

VME-MXI Problems

VME-MXI Problem HP VME-MXI Driver Module : National I

nstruments Products FORCE PowerCore 6603 or 6750 CPU

Modules : Power PC Tandra Universe or Universe II Chips :

PCI-VME Bus Bridge Chips CPU Module Halts and Never Restarts

Responding to the Reset Signal without Power is switched off

Improvements in ARCNET A HUB Box contains 3 ARCNET HUBs 33 HUB Boxes were Installed for 97 AR

CNET Lines Decreases 102/Day/Line ARCNET “Rec

onfiguration” Errors down to Zero

Configuration of the Test Bench

CP

U m

od

ule

VM

E-M

XI m

od

ule

Sy

ste

m M

on

itor m

od

ule

VX

I-MX

I Co

ntro

ller

MP

X m

od

ule

MP

X m

od

ule

RF

Vo

ltme

ter

CPU Modules tested

FORCE CPU40: MC68040, 33 MHz

FORCE CPU64: MC68060, 66 MHz

FORCE PowerCore 6603: PPC 603, Universe Chip

FORCE PowerCore 6750: PPC 750, 266 MHz, Univ. II

FORCE PowerCore 6750: PPC 750, 400 MHz, Univ. IIB

Normal and Abnormal Bus CyclesNormal Bus Cycle Abnormal Bus Cycle

Abnormal Bus Cycle

Normal and Abnormal DTACK* Signals

PowerCore 6603, with Universe Chip

Normal Abnormal

Abnormal DTACK* Signal

PowerCore 6750, 266 MHz, with Universe II

Normal Abnormal

Realtime Linux

Linux Disk I/O causes 20 - 30 ms of Latency,

possibly Latency can be around 100 ms or more.

Cause:

– Non-preemptive Kernel

– Scheduling Algorithm

– Interrupt Disabling

Non-Preemptive Kernel

Interrupt

Low Priority

High Priority

KernelLatency

Preemptive Kernel

Low Priority

High Priority

Interrupt

Kernel

EPICS iocCore

Network

Channel AccessChannel Access

Run-time DatabaseRun-time Database

Device Access LayerDevice Access Layer

I/O Buses (VME/PCI/ISA)I/O Buses (VME/PCI/ISA)

EPICS under L4-Linux

L4 real-time micro-kernel

XLinuxServer MEDM iocCore

Linux System Calls

Linux Server

Process

Interrupt

L4 Kernel ( IPC )

Linux Kernel

Process

Interrupt

Standard Linux L4-Linux

RT-thread Preempts Linux

Linux Server

RT- thread

Interrupt

L4 Kernel

Process

RT-thread Calls Linux

Linux Server

RT- thread

Interrupt

L4 Kernel

Process

What Happens if RT-thread Calls Preempted Linux?

Linux Server

RT- thread

Interrupt

L4 Kernel

Process

Latency