1

Device AccessThe Frontend Layers

CCT Workshop Preparation Meeting14. / 15. Apr. 2010

L. Hechler, U. Krause, A. Schwinn

2

AgendaAgenda

Part 1: Common Aspects• Device Access• Device Modelling Layer• Realtime or Device Control Layer

Part 2: Dedicated Systems• FESA• GSI Device Access• Similarities and Differences

3

Device AccessDevice Access

• SCADA Systems– I/O points, hardware-dependent values– Control device directly by writing set values to i/o points– Periodically read actual values– Quite a lot of system monitoring

• Accelerator Control Systems– Device abstraction: properties, physical units– Automatic device control triggered by timing system

4

Device AccessDevice Access

• Distributed system, device access via network– Persistent objects

• Uniform interface– Hide (abstract from) equipment specifics

• Hide network protocol specifics– from client applications– from frontend software

• Access rights

5

Device Access: FrontendDevice Access: Frontend

• Modelling Layer– Provide an abstract or idealized view of a device– Model device classes with properties

• A magnet has a field strength, a faraday cup has metering ranges

– Convert hardware-specific values into physical units• Field strength in Tesla, metering range in micro-ampere

• Realtime or Device Control Layer– Synchronize actions of distributed devices– Hardware-oriented view

• Hardware-specific values, channels, registers, ...

– Supervise hardware (online, interlock, ...)

6

Modelling Layer: API IModelling Layer: API I

• One device class => one interface

• Narrow interface– Device = object, properties via names– tk1mu1.read("FIELDS", data, ...)– Parameter 'data' has to be a versatile container

• Wide interface– Device = object, properties are methods or objects– stat = tk1mu1.status()– field = tk1mu1.fields().get()

7

Modelling Layer: API IIModelling Layer: API II

Device Access modes• synchron (blocking)• asynchron (non-blocking, needs callback)

• Monitor (connected to an event, needs callback)– Events: Value or state changed, threshold exceeded, external hw

trigger, timer expired, timing event (really necessary?), ...

• Subscription– Read consistent data from different devices and/or properties– Consistent data: set data from the same beam pulse (with the same

timestamp)

• Supervision of active monitors and subscriptions?

8

Modelling Layer: API IIIModelling Layer: API III

• Property– Timing independent– Timing dependent, needs VrtAcc or User number

– Set values should be writable and readable (FIELDS)– Actual values are (naturally) readable only (FIELDI)– But: POWER has no differentiation in S and I.

• Self-describing data vs. XML description– see Part 2

9

Modelling Layer: Super DevicesModelling Layer: Super Devices

• A super device is a composition of (real) devices– Profile grid plus collimator yields emittance measurement

• Cascading may be necessary (super super device)– sis(POWER) = off

• Two super devices may have the same real device.– Serial connection of magnets (one PS)

• A question of appropriate design, to implement this seamlessly

10

Realtime Layer: Realtime Actions IRealtime Layer: Realtime Actions I

• How to synchronize distributed devices?– Bind realtime action to timing event– Timing event x => Interrupt => RT-actions set/read values

simultaneously for all concerned devices

• Each realtime action is bound to an event– Timing event (from event bus) => write set value, read actual

value, ...– DRD, DRQ (from device) => data ready: read data from device;

data request: write data to device– Internal timer (from CPU) => e.g. timing event delay

• Action called in time or delayed (overrun)?

11

Realtime Layer: State Machine IRealtime Layer: State Machine I

Assure proper order of realtime actions• e.g. grid measurement: prepare -> start -> read• Use state machine• Some states are per VrtAcc• Identify sequence errors• Allows interlaced pulsed magnet control at HSI

12

Realtime Layer: State Machine IIRealtime Layer: State Machine II

power off

ready @ vrtacc x

prepared

@ vrtacc x

busy@ vrtacc x

error

Power on

Event preparemeasurement

Event startmeasurement

Event readmeasurement

Event preparemeasurement

13

Realtime Layer: Realtime Actions II Realtime Layer: Realtime Actions II

Gap actions• Non-realtime actions may nevertheless depend on event timing• e.g. switching power off in the middle of a vrtacc

– state machine may detect an error– a realtime action may be delayed

• Solution: execute non-realtime actions between two vrtaccs• Between two vrtaccs there's a gap of ~100ms • Gap is marked with gap start and gap end event, also command

events

• Unilac: No gap, special handling

14

Realtime Layer: In time vs. precisely timedRealtime Layer: In time vs. precisely timed

in time: rechtzeitig, precisely timed: zeitgenau

• A device must be set in time – before the beam arrives

– e.g. pulsed magnet in HEBT, t_extr – t_startcyc ≈1s

• A device must be set precisely timed – at the time of the event– e.g. to start ramp of SIS RF or SIS dipoles, t_jitter ≤ 15μs

• A CPU handles several devices successively– the last device must be handled in time– precisely timed handling only via common trigger, e.g. broadcast to

all devices

15

Realtime Layer: DataRealtime Layer: Data

• Set values must reside at frontend– no way to deliver or request them via network in time– one buffer per vrtacc

• Actual values reside at frontend too– one buffer per vrtacc, overwrite data on next event– ringbuffer, but also limited capacity

• Vrtacc-independent data– set values usually also at frontend, e.g. FIELDS of DC magnets– to resend value after RESET– actual values: it depends

16

Realtime Layer: Data Consistency IRealtime Layer: Data Consistency I

• Data exchange between modelling layer and realtime layer is asynchronous– set value

• modelling layer writes to buffer on client's request

• realtime layer reads from buffer at time of event

– act value buffer • vice versa

• Problem: – a buffer would have to be locked during write – no problem if modelling layer wants to read, it can wait– but realtime layer needs instant access

17

Realtime Layer: Data Consistency IIRealtime Layer: Data Consistency II

• Solution: synchronized double buffer– read from sync'd buffer– write into other ('unused') buffer– switch buffers when write is complete– switching has to be uninterruptible

– most buffers may be switched within beam cycle– some buffers must be switched during gap only

18

Realtime Layer: Device Control / SupervisionRealtime Layer: Device Control / Supervision

• Process requirements– actual value deviation– switch magnet settings (virtual accelerator)– start magnet ramps, RF ramps

• Equipment requirements– meet DRQ demands of function generators– closed loop control for SIS injection bumper, set value– react on interlock– count sparkovers per time, switch off device if maximum reached

19

Realtime Layer: Error Handling and ReportingRealtime Layer: Error Handling and Reporting

• Sophisticated error handling required– to quickly find the cause of a problem in the process– realtime software is hard to debug, even no 'printf' debugging– 50% of code on realtime layer is error handling

• Where to report errors to?– there's no direct client– one error buffer per realtime action– error ring buffers– send alarm– set error or alarm state

20

Realtime Layer: Where goes the timing?Realtime Layer: Where goes the timing?

• Generally to the realtime layer of the software– a CPU– is able to act accordingly– and - important! - react on errors (event missing, wrong order, ...)

• High precision triggers go directly to hardware– hardware must signal arrival of trigger– so software can handle missing trigger– => software needs timing too

• Payload data (time, Bρ, ...) must go to software

21

Part 2 – Dedicated SystemsPart 2 – Dedicated Systems

• FESA• GSI Device Access• Comparison DevAcc – Fesa. Similarities and Differences

22

Fesa: Hardware requirementsFesa: Hardware requirements

• CPU architecture– Any x86 processor– Any ppc3/ppc4 processor (at GSI the focus will be x86)

• OS– any Linux distribution for x86. Missing libraries can be added.– LynxOS for ppc4 (also Linux should work, but was not tested till

now)

SCU (pic from M.Thieme)

23

Fesa: Priority ManagementFesa: Priority Management

Server RealTime Server RealTime

(std::priority_queue) (posix ipc queue)

• split/unsplit deployment

Server: Manages all client requests (get,set,subscribe)

RealTime: reacts on events by executing „RTActions“

• thread–control via different priority levels• scheduling policity: RoundRobin (prio 0 to 99)• priority levels:

• Server: 25 (if not changed by user)• RealTime 30 to 70 +/- offsets• Background activities: 5

• Possibility to run the RealTime-part on a FPGA- softcore

To be evaluated

24

Fesa: Data accessFesa: Data access

• Const(configuration) data– per device– initialized from xml file on startup, never changed later

• persistent(Setting) data– per device– per cycle(virtual accelerator)– double buffered, buffer synchronized bevor execution of RTactions– field synchro-manager takes care of that (minimal mutex usage)

• Volatile(Acquisition) data– stored in ring-buffer of adjustable deep– per device

25

Fesa: Ringbuffer – Concept for AcquisitionFesa: Ringbuffer – Concept for Acquisition

va10 va05 va11 va01 va01 va13 va02 va02 va02 va07

Realtime

Server

4,7 5,6 1,3 9,7 9,7 1,0 3,5 3,5 3,6 0,7

Pro:- No double-buffer necessary (no additional “copy” of the data)- Possibility to ensure, that data can be obtained by the Server,

even if the same VA is played twice (or more)- Possibility optimize the buffer, according to the FEC–memory- Possibility to run with less memory than a double buffer. (can

be interesting for FEC’s with low memory)

Contra:- Not possible to ensure, that a get(…) will always find the requested VA- A minimal mutex protection to specify the index of Server and RealTime

process is necessary.

write

read

26

Fesa: RTActionFesa: RTAction

• comparable to EQM (Equipment Module) in the GSI FWK

• Connected to the GSI-specific timing implementation through the FesaGSI package.

• The dependency Event RTAction can be modeled in the instantiation document

• The dependency RTAction Property(s) can be modeled in the design

• „Event“ is not necesarily a timing event. The following eventsources do exist in Fesa3:– Timing (events from the timing system)

– Timer (a stop watch, which can be freely configured)

– Hardware (any kind of hardware trigger/interrupt)

– OnSubscription (If one Fesa class subscribes to another one)

– OnDemand (events, triggered from get/set actions on the server side)

– Custom (anything else can be implemented here by the user)

27

Fesa: ServerActionFesa: ServerAction

• comparable to USR (UserServiceRoutine) in the GSI FWK• run‘s within low priority• Manages all subscriptions within different priority levels for different

properties. (One can interrupt another)• Possibility to use self-defined get-action instead of default

implementation.• Notification Example:

TimingEvent RTActionnotify „status“

notify „flattop voltage“

ServerAction1

ServerAction2

Send new data to subscribers

RealTime

ServerIPC or local queue

28

Fesa: CollectiveNotification, EventConnectionsFesa: CollectiveNotification, EventConnections

• Collective Notification– Exists within Fesa, since it is a service which is automatically

provided by the rda-middleware.– Flush buffer is triggered after the notification, or after the

backgroundupdatequeue got empty

• Event Connection– Not available in Fesa3– Has to be „simulated“ by the cosylab Gateway, if old GSI

applications need to control a Fesa class.– Not needed to control Fesa3 classes

29

Fesa: DatabaseFesa: Database

• Fesa3 does not rely on any kind of database. It provides tools to use/fill a database, but it is not addiced to that DB. If you only want to generate a testclass, you dont need any database entry

• The following entrys propably will make sense to be stored/used in a DB– All properties of a class

• All fields of this property– All datatypes of these fields

– Which user/group has rights to read/write which property

– A table to show which device(Nomenklatur) can be found at which FEC(IP)

– Which user/group has rights to access which device

• After changing/adding/deleting propertynames/fieldnames, etc. in a fesa class, a tool will be in charge of (half)automatically update the DB.

30

Device Access: FrontendDevice Access: Frontend

• Modelling Layer: GμP with User Service Routines (USRs)– PPC with embedded Linux– asl321 with standard Linux– Windows XP

• Realtime Layer: SE (EC) with Equipment Modules (EQMs)– until today: nearly all device i/o via EC

• Communication with EC via dualported RAM (DPR)– minimal overhead, max. some CPU clocks

31

Device Access: EQMsDevice Access: EQMs

• Actions coded in functions: EQMs• Equipment specific actions

– Send set value– Read actual value– . . .– May be tricky, e.g. several FGs with different interpolation distance

• System actions– Look for connected hardware– . . .– May be overloaded by equipment specific EQMs

• Flexibility: Standard actions where possible, specifcs where needed

• EQMs deal with sometimes challenging hardware configurations

32

Device Access: Equipment Configurations IDevice Access: Equipment Configurations I

EC EC EC

One Device

one HW-IFOne Device

several HW-IFs

Several Devices

one HW-IF

33

Device Access: Equipment Configurations IIDevice Access: Equipment Configurations II

• Magnet TK1MU1: One device, one power supply, one hardware interface

• Kicker S04MK1E: One device, 18 high voltage modules, 28 hardware interfaces

• Profile Grids UL5DG5, UL5DG6, UL5DG7: Three devices, one electronic, one hardware interface

• Additional challenges with grids: – Grid may spread over different mux channels– A mux channel may hold different grids

34

Device Access: TimingDevice Access: Timing

• Standard timing event– (PZ-ID), VrtAcc, Event

• Gap events– gap start, gap end, command

• Date and time information– old: hour, minute, second, 1/100 second– new: UTC since 1970

• ECCs – Event Connected Commands– kind of broadcast commands to all related ECs– used for therapy mode

35

Device Access: Interlaced HSI-Magnet Control Device Access: Interlaced HSI-Magnet Control

36

comparison EQM - RTActioncomparison EQM - RTAction

• DevAcc - EQM– Event => Interrupt => EQM is a real ISR– No interruption of EQM, since no higher priority interrupts exist– Clock interrupt is disabled!– Simultaneous service of devices throughout accelerator within 15μs

• Fesa - RTAction– HWEvent => SWEvent => RTAction– could be interrupted by other RTAction with higher priority, if defined

in the design

37

comparison access rightscomparison access rights

• Fesa– check on property level (not yet implemented)– access right table in the oracle database

• Device Access– each property (per equipment model): criticality level

• free access, device modification, system, critical

– users: rights granted for devices (device groups)• right levels: read, modify, localsystem, system, administrator

• access to corresponding criticality level

38

comparisoncomparisonFesa rdaData – Device Access AccDataFesa rdaData – Device Access AccData

rdaData - container AccData - container

• Get(prop,cycle,data,context)• Set( … )• MonitorOn(…)• MonitorOff(…)

usage:

1. dev = getDeviceHandle(nom)

2. dev->Get(…)

RDA-Middleware(FESA) GSI-Middleware (DevAcc)

• bool• signed char(byte)• double• float• long int• long long• short• char*

• array or single• each element has a *char „tag“• Insert/extract methods for

each datatype• Self-descriptive

• read(prop,vrtacc,data)• write(...) / call(...)• requestRead(.., cb) / cancel()• connectRead(.., cb) / disconnect()

usage:

1. dev = createDeviceReference(nom)

2. dev->read(…)

• [un]signed byte• [un]signed word• [un]signed long• [un]signed quad• float• double• string

• array or single (not string)• structure (mixed types)• Insert/extract methods for

each datatype• Not self-descriptive

39

Fesa: data container - rdaDataFesa: data container - rdaData

• Generally:– Information about all properties, fields and data types of an operational

Fesa class will be availabe in a database.– „self descriptive“ for the rda Container means:

• the container has information about the data-type of each element

• Each element comes within a „tag“, which brings more information about the data, like „voltage“, „frequency“ in Fesa: tag = fieldname

• If you want to get a console - output of all data inside an rdaData element, you just need to execute data.print()

• Acquisition:– Fesa Client: Always self descriptive.

• Setting:– Client Fesa: self descriptive as long as client fills the „tag“. If the FWK

relies on a specific tag, the client will receive an error, when performing the set-action

40

Device Access : data container - accDataDevice Access : data container - accData

• Formal description (XML) <property category="slave">

<name>CURRENTI</name>

<description> Current actual value in Ampere. </description>

<action type="read" access="free" medlock="none">

<data>

<value type="Float32" name="currenti">Current actual value.</value>

</data>

</action>

</property>

• Generate adapter classes– getter/setter, names same as data field names

• Usage AccDevRetStatus ReadCURRENTI::read(SLong vrtAcc, const AccData& rcvPara, AccData& sndData,

AccStamp& stamp, AccEFICD& eficd)

{

ReadCURRENTIDataP dataP = ReadCURRENTIDataAdapter::createSndP(sndData);

dataP->currenti(...)

return setCompl(primStat, secStat, "Read CURRENTI");

}