software tools for mps kajetan fuchsberger mpp workshop, 2013, annecy many thanks for input from m....
DESCRIPTION
3 Kajetan FuchsbergerMPP Workshop Annecy, , Software Tools for MPS Content Tests & ProceduresEarly detection of failuresAperture MeterTRANSCRIPT
Software Tools for MPSKajetan Fuchsberger
MPP Workshop, 2013, Annecy
Many thanks for input fromM. Zerlauth, J. Wenninger, R. Schmidt,
G. Kruk, V. Baggiolini, G. Papotti, D. Jacquet and the TE-MPE-MS Software Team
Software Tools for MPS
The solution for everything ;-)
3Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Content
Tests & Procedures
Early detection of failures
Aperture Meter
4Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Content
Tests & Procedures
Early detection of failures
Aperture Meter
5Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Tests and Procedures Currently: Sharepoint Sites
Disadvantages: Order not enforced
(scheduling done ‘on the fly’).
Nothing enforces the tests to be really done.
No real overview, what was done, what still has to be done?
6Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
The AccTesting Framework Designed from the experience
of Hardware Commissioning.
Successfully Used in HWC Campaigns 2012, 2013
Enforces Correct Order of Tests
Many additional features: Automatic Scheduling Constraints Statistics
7Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
AccTesting Concept
Migrate of MPS Commissioning Procedures: 1) Transform all the MPS comissioning steps to
AccTesting (simple sign-only tests). 2) Replace one by one with automated tests.
Test-Phases.
Tests
8Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Barriers When the commissioning of a system
reaches a Barrier Point, it has to wait until all other systems (which have the same barrier) reach it also.
QPS IST1
BLM IST2 IST3 IST4 INJ1 INJ2
BIS IST5 IST6 IST7 INJ3
LBDS IST8 INJ4
RQTL… IST10 IST11 POW1 POW2
Ready for Powering
Ready for Injection
Knowledge of System-Dependencies required!
9Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Interlocks based on Test Plan? Examples:
Prohibit, that circuit can be powered, before all the tests are done. Do not allow injection of beam before all necessary MPS tests are
done.
Possibly: Enforce Tests after changes: E.g. Which tests have to be executed, after a change of a QPS – card? Pre-defined test plans or (partially) auto-generated? Knowledge about System-Dependencies required!
What to interlock? E.g. Additional Circuit lock on PIC level which allows tests? Injection interlock?
10Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Automation? Automate whatever can be automated!
(Avoid Errors, Reproducibility)
First steps: Concentrate on systems, where many interlocks of same type: BIC/PIC: Already dedicated tool, in the pipeline to be integrated with
AccTesting Collimators
Integrate BIS connection Test? Do we really have to look ‘by eye’ on lossmaps? Could there be a Hierarchy-check without a reference?
Vacuum BLMs:
E.g.: Test Of Latency: Close collimators: Done in pt3, pt7: Could be done for more, if automatic.
Others?
11Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
System Dependencies & Information
Key Concept: Use existing Sources!
Prototype in place
Possible Cooperation between TE-MPE-MS software team and BE-CO-DO (S. Jensen, D. Csikos).
System Information
Server
System Providers:• Lsa• Layout Db• …
Dependency Providers:• Layout Db• Runtime info (JMS)• Java code analysis• …
Information Providers:• Status• Issues (Jira)• Deployments • Faults? …
Clients (e.g. AccTesting, Diamon, ….)
12Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Test Analysis Vision:
Simple description of test-expectations (assertions) on signals resulting from tests (Java).
Universal GUI components to visualize problems.
First Implementation: Cooperation between TE-MPE-MS software team and BE-CO-DA (R.
Gorbonosov, A. Jalal) Used in AccTesting could be used e.g. in future PM modules. (Maybe also in sequencer
tasks and other checks?)
13Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Data Analysis TE-MPE-MS Vision: More general Data Analysis Framework
Key Concept: Perform analysis as close as possible to the data!
(Potential) Collaboration with BE-CO-DA to optimize resources, avoid duplication of efforts
First Implementation in place.
Analysis Framework (Cluster?)
Logging DB
Hadoop Cluster?
Post MortemClients (e.g. AccTesting, Timber, Custom Analysis….)
Others?
14Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Content
Tests & Procedures
Early detection of failures
Aperture Meter
15Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Early detection of FailuresYASP
LSA Server
trim
PcInterlockGUI
PCPC
PC
trim subs
crib
e
SIS
monitor
Dump!
?
?Not Allowed!
PcInterlock Server
16Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
LSA: The natural place to check
LSA Server
Applications: YASP Lumi-Scan
application Others…
Checks: Power Converter
Currents Collimator Settings BPM readings (actual +
predictions) online modeling?
Others?
17Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
LSA checks - proposal Vision:
Have an API like ‚isAllowed(TrimRequest request)‘
First implementation: Use ‚TrimPostProcessor‘ mechanism of LSA. Executed after trim saved in DB, on error LSA will do
a rollback. Incorporation = Trim Same mechanism to avoid
propagating trims, which would otherwise e.g. fail during ramp.
Override for MDs.
18Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Further securing LSA LSA Method protection
Currently: Only basic protection (Sending of timing events, Trim, Drive)
Every method in LSA should be reviewed, if it can do harm or not and protected appropriately
Protect reference Cycles for PcInterlock/SIS: First approach: TrimPostProcessor which evaluates
RBAC roles
19Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Content
Tests & Procedures
Early detection of failures
Aperture Meter
20Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Aperture Meter
21Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Aperture Meter - Status Prototype Operational
Proved to be very useful during MDs Follows the operational cycle (optics, BP, time in BP) Listens to (some) LSA trims
What is missing? Improvements of User Interface Performance Improvements Tighter integration with Operational changes
(Collimator offsets after alignment, BPM usage from Yasp) Could be used as another source for LSA trim-verification
(e.g. large collimator movements, Collimator Hierarchy)
22Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Summary MPS Commissioning:
Tracking in Acctesting Framework. Automate! (Step by Step) Candidates?
Preview: Two new frameworks: Tracking of system dependencies (Test-) Data Analysis
Early detection of potential problems could be done in by additional trim checks in LSA.
Aperture Meter: Useful prototype, but needs improvements.
23Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Tools, Tools, Tools, Tools, Tools, …?
…Tools do not solve everything! Culture & communication
(High Quality) Software is not for free!
Challenges: Large part of Software manpower goes already into maintenance. A lot of ‘grown’ projects, partly written by un-experienced programmers
(E.g. Students).
How to improve? Reliability = Maintainability = Quality
(Self explaining code + Automated Testing, Testing, Tes…. ) Supervision of any SW projects from two sides: System Expert + SW expert
(‘Mentoring’) Who has the overview and can judge what is already out there and into
which framework a certain requirement would fit best? Communication!
Who is authorized to define priorities? Resources?
24Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Thank you for your Attention!
Questions?
Small Steps-
Big Visions!
25Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Fail Fast! Actual BP:
Would a certain trim exceed some limits?
Check before functional BP: Collimator-limits prediction Corrector Interlocks vs. Lsa settings Check at incorporation time
26Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Single Source for „Operational State“
For certain applications BeamMode is not enough!
Combination of: Beam Mode, Beam Process, Time within BP, (Optics)
Currently (at least) 2 implementations: Combination of Beam Modes and PC state
changes(Aperture Meter)
Dedicated timing events (PcInterlock) Both not optimal. Reference Implementation
required.
27Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Fail Fast!
„A fail-fast system is designed to immediately report […] any failure or condition that is likely to lead to failure. ”
[Wikipedia]
28Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Data Analysis Mandate in BE-CO-DA (C. Roderick) to provide Data Analysis Solutions
Currently in Use Case Analysis Phase
Aim to be as easy as possible for users to specify: What to produce for an analysis case (e.g. signal names, units, description) What are the inputs (signals and sources) Which algorithms should be applied When analysis should be performed (event driven e.g. fill / beam mode, time
interval driven e.g. every 15 minutes)
Without user needing to: Configure a specific development environment Care about compilation and deployment Look elsewhere for signal names etc. i.e. provide directory of signals and sources
Key concept: Perform data analysis as close as possible to the data!
29Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
MPS Commissioning
What is missing? Dependencies between Systems (e.g. ELQA for a circuit
is ready, when the test are done for all magnets within the circuit)
Global Phases?
QPS IST1
BLM IST2 IST3 IST4 INJ1 INJ2
BIS IST5 IST6 IST7 INJ3
LBDS IST8 INJ4
RQTL… IST10 IST11 POW1 POW2
IST Powering Injection
30Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
System Dependencies Vision: Central Instance (Server?) to answer questions like:
Which magnets are connected to a certain power converter? (AccTesting – ELQA) What is the common point for a set of crates? (Diamon) What systems have to be potentially re-tested if e.g. a QPS card is changed?
(AccTesting) What systems are potentially affected by a OS upgrade on a certain host? (SUWG)
31Kajetan Fuchsberger MPP Workshop Annecy, 2013-03-13, Software Tools for MPS
Interlocks during functional BPs
In place for orbit correctors. Ready to be extended to other circuits.
Potentially useful for other systems, e.g. Collimator movements?