stefan koestnertutorial – user guideseptember 2007 tell1 - control guidelines for users &...

Stefan Koestner Tutorial – User Guide September 2007

TELL1 - CONTROL Guidelines for Users & Developers


Contents:

• framework components (fwCcpc, fwHw, fwTell1)• configuration of Tell1 boards• finite state machines• design issues - special requirements• user specific - what you still have to do


fwCcpc – in a nutshell:

• interface between PVSS and CCPC using DIM and a generic server

• basic test panels & PVSS library

doxygen documentation and tutorial under:http://lhcb-online.web.cern.ch/lhcb-online/ecs/PVSS_CCPC/default.html

http://lhcb-online.web.cern.ch/lhcb-online/ecs/PVSS_CCPC/default.html

http://lhcb-online.web.cern.ch/lhcb-online/ecs/PVSS_CCPC/default.html


fwCcpc – in a nutshell:

• communication via Ccpc DP• each Tell1 has its own DP and each bus has its own DP element

• Note: use fwHw for accessing registers and not this DP! (model your hardware)


fwHw – in a nutshell:

• a tool (panels & library) for modeling hardware as PVSS DPs and creating recipes. generic: SPECS & CCPC


fwHw – in a nutshell:

• each register has its own DP and DIM srvcs/cmnds associated

• subscription: - register settings transferred to ccpc and stored there in list. - create DIM server cmnds/srvcs.

(establishes communication between server & PVSS)

doxygen documentation and tutorial under:http://lhcb-online.web.cern.ch/lhcb-online/ecs/FwHw/default.html

http://lhcb-online.web.cern.ch/lhcb-online/ecs/FwHw/default.html



Recipes – in a nutshell:• a recipe is a collection of registers, whose

values can be stored in the configuration DB - from where they can be applied to hardware at any time.

• Panels and framework functions to create and operate on recipes are provided.

documentation under:http://lhcb-online.web.cern.ch/lhcb-online/ecs/FwHw/default.html




fwTell1 – in a nutshell:

• datapoint representations of Tell1 types (including detector specific parts)

• standard recipes of all recipe types (including detector specific part) To be imported from hardware: myTell1_’type’

• panels for device units (stopped counting ) – adopting automatically to Tell1 type

• control unit with user interface• finite state machines


fwTell1 – in a nutshell:

• use fwHw tool to instantiate your Tell1 boards (instantiate datapoints)

• use DEN (device editor/navigator) to associate device unit with datapoint

• create your FSM tree

screenshot of Device Editor/Navigator


fwTell1:Tell1 control unit:

pof – file:stores the firmware of FPGAs

cfg – file:server can access EPFL C-codeto configure the boards

each board has its own ccpc


fwTell1:Tell1 control unit:

Commonly modified configurations can be sent to enabled boards

Send down commandsto enabled boards.(configure uses recipes e.g. “PHYSICS\Configure”


fwTell1:Tell1 device unit:

A tab register is providedthat allows you to access the common part of eachTell1 board.

A tab for specific part is foreseen.

The panel configures automatically according to board type.

Some checks are performed.

Watch the info buttons andtooltips for documentation.

And many registers to be monitored … (a few hundred)



in fact a lot of panels …We provide tool-tips and info buttons for help!And perhaps a dedicated tutorial on device unit panels.



This is whereyou can put yourown panels!


fwTell1:Tell1 device unit: We are not providing

the monitoring of each sub-detector specificparameters. (but a few…)



We allow for spying on thedata of the MEP buffer.

The structure of the MEPbuffer differs – we do not provide parsersfor each sub-detector.



Some examples for parsingthe MEP buffer…



We divide between 3 sub-recipes.Each sub-recipe has its ownpanel.

You can export/import recipesor simply read them directly fromhardware.


fwTell1:

Define your specific recipe type.Create the appropriate panels.

specific recipes:


fwTell1 – conclusion:• for commissioning you can just use the common

part• BUT: start developing the specific part NOW!

• NOTE: the specific part is related to your firmware. Get in touch with EPFL Lausanne before you start! You might profit from already existing things.

doxygen documentation and tutorial under:http://lhcb-online.web.cern.ch/lhcb-online/ecs/PVSS_TELL1/default.html

http://lhcb-online.web.cern.ch/lhcb-online/ecs/PVSS_TELL1/default.html

http://lhcb-online.web.cern.ch/lhcb-online/ecs/PVSS_TELL1/default.html


Ctrl managers – in a nutshell:

• fwCcpcScript.ctl: connects to running servers, automatic register subscription (crash/restart)

• fwTell1FSM.ctl: needed to stop monitoring of registers when switching panels in DUs

• fwTell1DisableArchives: disables the archive bit of registers gain speed for configuration

• fwFSMServer


Configuration of many boards:

• trials and errors• benchmarking results


Configuration of many boards:A first impression of scaling:

a snapshot from the talk of June 2007:

50 boards ~ 6 minutes


first steps …

black: old - no changes applied

yellow/yellow: disabling archiving (but 35% CPU load from panels) lpink: disabling archiving & 1 ctrl manager per DU (still 35% CPU load from panels) green: disabling archiving (no CPU load from panels) red: disabling archiving & 1 ctrl manager per DU (no CPU load from panels) blue: reference when applying just specific part of recipes (disabled & 1 ctrl/DU)

50 boards ~ 3.5 minutes

• disabling archiving significant• additional ctrl managers in FSM significant under high load• panels consuming much CPU?


same as before – two (lower) lines added …

black: checks on serverside - comparison of applied values with hardware values read back

cyan: common part of recipes (16 registers of 32 bit)

50 boards ~ 2.5 minutes

•changes on server significant but still too small

complete recipe of type ST: • 236 registers• ~80 kB

grouping and packing together memory blocks would help, but not so easy to implement in FPGAs


so where does the time go …

• most of the time consumed by DIM, Event and Data manager• in current design each register is sending a DIM command


parallelizing helps …

… but not yet enough!


new approach – without changing the hardware representation …

getApplyRecipes() for fast Tell1 configuration:

• retrieve recipes from configuration DB• pack them together in one buffer• send them as one buffer/DIM command • decode it on server

http://de.wikipedia.org/wiki/Bild:Johann_Heinrich_F%C3%BCssli_018.jpg


\0 &… …

C& C &…

\0

The buffer - array of unsigned char:

# registers register-names size of registers

data checks buffer corruption

…

command character


\0 & …& C &

on the server side:

parsing through buffer

• get number of registers• get strings, compare them with list• fill array of pointers to list of registers (all hardware info stored) note: subscription is necessary!• check for corruption• fill array of register sizes• check for corruption• execute commands with appropriate data• final check


the results for getApplyRecipes():red: one DIM command per register (old design) – but improved framework

blue: sending one single buffer containing all registers (new design)

grey: applying .cfg file

1.5 sec/board 1 sec/board no slope

50 boards < 1 minute

most time gets lost when retrieving the recipes from DB storing the whole buffer?I am optimistic that we could come closer to the .cfg file!


how to create recipes without typing several thousand parameters:

• ‘get image from current board settings’ reads registers from hardware and saves them as a recipe(you can upload the .cfg file before)

• ‘import recipe from other hardware’ takes the recipe previously defined for another hardware.(you can use ‘myTell1_type’ as default recipe)


Conlcusion:• creating recipes is not a big deal (tools are available)• downloading recipes to hardware is already fast & sufficient• we still support the interface to the .cfg files (original C-code)

Note: some parameters are providedby Gaudi jobs. interface under development


Finite State Machines:

towards a fully automatized expert system …


Tell1 boards belong to the DAQ domain.Predefined transitions and states:

FSM – in a nutshell:

Recipes are applied under ‘Configure’ –transition from NOT_READY to READY.

Checks are done to verify the statusof the board.If they fail status moves to ERROR.

If connection to ccpc is lost, it moves to UNKNOWN.


FSM – in a nutshell:

• functions to be called by FSM are provided in fwTell1.ctl

• if needed you can provide a specific function to be called by the FSM (e.g. for calibration) fwTell1_Muon_Cmnd(string domain, string device, string command, string parameter)

Please read Integration Guidelines:https://edms.cern.ch/file/732486/3/LHCb_ECS_Guidelines.pdfand FSM Guidelines:https://edms.cern.ch/file/655828/3/LHCb_ECS_FSM_Guidelines.pdf

https://edms.cern.ch/file/732486/3/LHCb_ECS_Guidelines.pdf

https://edms.cern.ch/file/655828/3/LHCb_ECS_FSM_Guidelines.pdf


FSM – from a higher point of view:

OTADCS

OTAHV

OTADAI

OTADAQ

INF DCS HV DAI DAQ L0 TFC HLT LHC

OTALV

OTATEMP

OTAFEE

OTATELL1

OTACRATES

OTACOOLING

OTA OTC

OT

Templates available:

Please use them as they are

Contact Clara for integration issues!

Storage

ECS


FSM – Run Control:• Run Control is like a

Matrix, can exclude/include:– by SubDetector part– Or by Domain

• “Standalone” run, need to allocate:

• TFC• HLT• Storage• Monitoring

• Allocate uses the Partition Settings

• TFC, HLT, Storage, etc. will be ready soon.


FSM – Run Sequences:

1 - Configure/RUN_TYPE=“CALIB_TIMMING”1’ - Configure/RUN_TYPE=“CALIB_TIMMING”/N_STEPS=102 - READY3 – Start4 – RUNNING5 - Start/N_TRIGGERS=100006 - RUNNING7 - TFC Paused (after N_TRIGGERS)8 - Step/RUN_TYPE =“CALIB_TIMMING”/STEP_NR=19 - RUNNING10 - Continue/N_TRIGGERS=10000

OTA OTA_DAQ TFC1

1’

234

5

67

89

10

N S

tep

s

OTADAQ

OTATFC

OTAFEE

OTATELL1

OTA

OTADev1OTADev1OTADev1OTADevN

OTADev1OTADev1OTADev1OTATell1N


FSM – Run Sequences:• The Activity can be

saved, contains:– Nr Triggers

– Nr Steps

– Nr Sub-Farms, etc.

• It corresponds to the RUN_TYPE that gets propagated down and used to form a recipe name by all devices.


FSM – Run Sequences:• Configure Command

– As usual propagates down so that each device can apply a recipe (or more) based on:

– RUN_TYPE (and N_STEPS)

• Step Command– Again propagates down, should be accepted in

state RUNNING (and return RUNNING) • Each device should change only the necessary

values based on:

– RUN_TYPE and STEP_NR


FSM – Conventions:

• Recipe Names:– Guideline: <Run Type>/<FSM Command>, ex.:

» PHYSICS/Configure, PHYSICS/Start» CALIBRATION/Configure, etc.

• Necessary because a Configurator Object will preload recipes from DB to cache


• started with a generic/altruistic server

• more and more intelligence moved to server side

few comments on design:


clear separation between hardware and control issues – a ‘smarter’ server:

• server keeps track of all the information about the register (e.g. type, address,…) and stores it in a list• server treats each register in the appropriate manner(e.g. Fifo vs. standard blockwrite or different I2C derivates)• Once the register is defined the control system (PVSS) knows nothing about the hidden details and just writes to or reads from a register less and more readable code & better performance (interpreted language)

server-side PVSS-side

Intelligence related to hardware Intelligence related to ECS


how to keep the design open:

• a ‘commonOperation’ part of the Ccpc DP allows to send a string to the server. Anything can be done.Identifier and parameters are separated by comas.

• a ‘user specific’ register type which offers many parameters to be filled in inside the fwHw tool. Any type of register can be modeled.

• the new recipe block executed via ‘configurationOperation’ on the Ccpc DP is in fact a set of instructions, which in addition can be stored in the Configuration DB.

\0 & …& C &

commands


What is left for the sub-detectors?in fact not much …

• Specific part has to be inserted into datapoint type• The panels related to the specific part: - MEP buffer spy - Monitoring of specific registers/values - Recipe related things• specific functions to be called by FSM (e.g. for calibration) preferably packed as a library (obey to naming conventions)• Scripts inside device units might have to be extended

… but still a bit!

stefan koestnertutorial – user guideseptember 2007 tell1 - control guidelines for users &...

Documents

tell1 device unit

hardware slide

physicsconfigure slide

tell1 control unit

device unit panels

tell1 type control unit

specs ccpc slide

mep buffer slide