Slide 1

Hot Checkout System for Accelerator Operations at JLab Ken Baggett (Team Leader) Theo Larrieu Ron Lauz Randy Michaud Ryan Slominski Paul Vasilauskis

Slide 2

HCO Training Session Presentation Objectives: Define hot checkout (HCO) Define HCO system roles and responsibilities Explain the use of the new HCO software tool Detail component signoff requirements and notifications Present checklist requirements for HCO components Software Demonstration Q&A 2

Slide 3

Hot Check Out (HCO) Definition Hot Check Out: The process by which all required hardware and software systems (old and new) are tested for proper functionality and signed off as being ready for beam operations. 3

Slide 4

Responsibilities Technicians/Engineers Work to verify operation of components based on HCO checklist requirements. Sets status to Checked. Group Leaders (or Group Supervisors) Work with PD to determine and sign-off on system readiness. Sets status to Ready. Geographic Integrators Responsible for the readiness of their geographic area. Works with affected groups and Commissioning Team to assure HCO compliance for beam destinations Management Program Deputy, Operability, Geographic Integrators, and 12GeV Project monitor/drive HCO progress based on beam destinations. Operations Verifies component status based on beam destinations prior to establishing beam for the Program Deputys accelerator program. 4

Slide 5

Getting to the HCO Tool https://accweb/hco/ Login required to edit 5

Slide 6

Hierarchy of Categories The status of each node in the tree structure is determined by its child nodes: The status of each category is determined by its systems, each system status is determined by its subsystems, each subsystem status is determined by its components, and each component status is determined by its responsible groups. 6

Slide 7

Status Options Not Ready Default setting at the beginning of the HCO process. No HCO has been performed. Checked Component has been verified by a Technician/Engineer as fully functional. Ready Component has been verified by Group Leader as being ready for operations. Masked - HCO was not completed on the component AND determined to be not required for the scheduled program. Only Operations Department Head has authority to Mask. 7

Slide 8

Overall Accelerator Readiness Screen Displaying all systems status is primarily used by the PD and Operations to view overall accelerator readiness. However, groups can view their systems readiness by filtering. 8

Slide 9

Filtering by Destination Filtering helps get the list of items to a manageable level. Filtering on Beam Destination displays all the systems requiring HCO for beam operations to the selected destination. 9

Slide 10

Filtering by Group Used primarily by groups ( Vacuum, RF, DC, ICN, Alignment, etc.) to determine what components they are responsible for determining readiness. 10

Slide 11

Filtering by Region and Group By further filtering by Region the list narrows even more. Very useful when performing HCO in a particular area of the accelerator. 11

Slide 12

Signoff Screen Must login to change status AND a Group and Subsystem must be selected List of Subsystems is based on the selected Group Optional Filtering Regions (several can be selected) Status (Not Ready, Checked, Ready) Ready Turn (YES indicates previous Groups have signed off and components Are waiting for your signoff) Selected Group is outlined with a box to help eliminate signoff errors 12

Slide 13

Signoff Screen (cont.) Three ways to select items for sign-off Single click on the item Multiple use Ctrl click All Use symbol THEN Push Edit Selected Choose desired status Enter a comment Push Save. 13

Slide 14

Signoff Rules Users can only signoff on one group at a time A status can only be set to Ready if it has already been set to Checked Only authorized Group Leaders can set the status to Ready A comment is required if a status change will cause a cascading downgrade of other groups readiness 14

Slide 15

Group Order Setting status to Checked can occur in any order but setting status to Ready MUST be done in the order listed. If a group downgrades their Ready status for a component then all downstream groups for that component will revert to Checked. If, in the table below, MCA1L01B magnet is removed by the Installation Group AND set to Not Ready, the downstream groups Vacuum, Alignment, and DC Power will all automatically reset to Checked. At that point each of the three Groups has the choice to reset their status to Not Ready or leave it as Checked. 15

Slide 16

Daily e-mail Alerts The HCO system is set to automatically send Group Leaders a report e-mail at the end of every day. Signoff Progress Report group component status counts. Signoff Activity Report activity in the past 24 hours including any cascaded downgrades. AGAIN: The Group Leader is responsible to determine what action is necessary for cascaded downgrades (reset their status to Not Ready or leave it as Checked). Groups Leaders Signoff Report indicates components waiting for the Group Leader to set to Ready. 16

Slide 17

Checklists Checklists are NOT procedures but are used to systematically verify system status. They are NOT a learning tool; Thats what procedures are. Checklists come in various forms; Read-Do Read a step and perform an action Do-Confirm Perform a set of steps then document them as complete Data Taking When steps require taking data. These checklists become permanent records and must be recorded as such. Checklists can be developed by anyone but are not considered Published until signed off by the appropriate Group Leader. The master checklist is stored, and revisions tracked, in the HCO system database. 17

Slide 18

Checklist Control Each checklist requires a digital signature (pushing the sign and publish button) by an authorized Group Leader before the checklist will be published to general users of the HCO software. New revisions to an existing checklist require repetition of the sign & publish process. Group Leaders can make changes without unpublishing the document; others cannot. The Group Leader must unpublish it before others can make changes. Then the Group Leader must publish it again. New revisions must include a brief comment indicating changes since the prior revision. Prior revisions will be maintained in the database. The database will be able to report checklists whose publication date is older than some TBD threshold. Checklists beyond that date will not be accessible until they have been reviewed and uploaded to the database. It is the responsibility of the Group Leaders to maintain completed checklists. 18

Slide 19

Checklists Tool Checklists are generated using the HCO Checklist Tool. They can either be built using the tool or placed there using the copy/cut and paste functions of other documents. 19

Slide 20

Example Checklist 20

Slide 21

Links, Reports, and Help Tabs Links Tab contains links to the following: ABIL (Accelerator Bypassed Interlocks Log) e-Log (electronic logbook) ATLis (Accelerator Task List) Others as necessary Reports Tab can be used by Group Leaders to determine their systems readiness. Help Tab presently includes contact information for HCO assistance and a link to this training document. 21

Slide 22

HCO Process Execution Initiation of the Program Verify ABIL by System Program Deputy sets the initial beam destination HCO software will populate a list of components that require HCO based on the selected destination Group Leaders verify that the lists of components are accurate and notifies the HCO Team if changes are necessary (additions and/or deletions) Coordination of group needs and desires HCO Team (PD) leads the daily 0800 meetings to Refine the HCO plan for the day Communicate system status Resolve any group and/or schedule conflicts 22

Slide 23

Schedule Training 8/07/13 (live session) Technicians/Engineers, Group Leaders, Geographic Integrators, Operators, and Managers Web-based training available 8/16/13 HCO software goes live 8/19/13 Call for ABIL compliance 8/27/13 - 9/13/13 Group Leaders ensure remaining interlock bypasses are properly documented AND justified Hot Checkout begins 9/16/13 PD is point of contact for HCO Daily HCO meetings begin (0800) Geographical Integrators work with groups and Commissioning Team to assure HCO compliance for selected beam destination HCO Team will meet weekly during HCO to determine if the process is advancing as expected and act accordingly HCO Complete 10/29/13 23