KISWG 10/29/2003 1

KECKINTERFEROMETER

ORR Readiness

Mark Colavita & Peter Wizinowich

KISWG

Pasadena 10/29/2003

KISWG 10/29/2003 2

KECKINTERFEROMETER

Question to Address (2/3/2003 pre-ORR presentation)

• At the ORR– Will the instrument be capable of achieving the requirements in the

PLRA at the start of routine science operations? (V2 mode)– What are the relevant performance metrics and current performance?

(V2 mode)• Today

– What are the specific performance requirements, and how do they relate to the PLRA?

– How will compliance be demonstrated?– Are they technically achievable?– What is the current performance?– What is the plan to get from current performance to ORR? Is the plan

realistic?– What are the risks for routine science operation? How will they be

mitigated? Are there additional risks not yet identified that need to be addressed?

KISWG 10/29/2003 3

KECKINTERFEROMETER

Specific Performance Requirements (2/3/2003 pre-ORR presentation)

• PLRA* requirements for V2

– System Performance Goals

» Provide 5% rms accuracy V2 amplitude measurements for targets as faint as K=9 using the two Keck telescopes

» Provide at least 90% uptime for the interferometer and 90% uptime for the two 10m telescopes and their AO systems

– System Performance Requirement

» Provide at least 90% uptime for the interferometer and 80% uptime for the two 10m telescopes and their AO systems

*Program Level Requirements Appendix to Navigator Program Plan

KISWG 10/29/2003 4

KECKINTERFEROMETER

Pre-ORR Testing from Oct Observing Run

• Targets included measurements of well-characterized PTI binary

– Used neutral density filters to attenuate this bright star to simulate K=9 ORR requirements

» Data accuracy still being assessed

» Observation

• K=9 spec will need to be qualified with a V or R magnitude, as stars faint to AO are less well corrected, and provide less flux into fringe tracker

–(Fringe tracker measures flux * Strehl)

• Plan to provide this for ORR

– Also tests of fringe tracker low-V2 tracking limit

KISWG 10/29/2003 5

KECKINTERFEROMETER

Oct Data(Most processing still in-progress)

KISWG 10/29/2003 6

KECKINTERFEROMETER

Prior (May, Jan) accuracy tests

KISWG 10/29/2003 7

KECKINTERFEROMETER

Other tests

KISWG 10/29/2003 8

KECKINTERFEROMETER

Provide at least 90% uptime for the interferometer and 90% uptime for the two 10m telescopes and their AO

systems (2/3/2003 pre-ORR presentation)

• Quoted directly from the PLRA

– NB: not a measure of “open shutter time” or efficiency, which is actually quite low for any instrument, like IF, that is calibration-limited

– A measure equal to 100% minus the downtime required to fix IF hardware or software problems during scheduled science observing

• Telescopes/AO addressed in a few slides

• Demonstrating compliance

– Mostly proper bookkeeping

» Identify a timekeeper to track system state vs. time

» Demonstrate compliance on two runs prior to ORR with good weather and median or better seeing

KISWG 10/29/2003 9

KECKINTERFEROMETER

System uptime & scans per hour

0

1

2

3

4

5

6

7

8

9

10

Jan-03 Feb-03 Mar-03 Apr-03 May-03 Jun-03 Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04

avg scans /hr

peak scans /pr

ORR goal

ins t uptim e

fac uptim e

ins t req/fac goal

fac req

Inst. uptime req. / fac. uptime goal

Fac. uptime req.

Scans/hr goal

Peak

Avgs

KISWG 10/29/2003 10

KECKINTERFEROMETER

Uptime summary• Interferometer uptime 90% 2/5 runs• Facility uptime 90% 1/5 runs; 80% 3/5 runs• October run problem areas

– Major» Telescope control system crashes

• K2 continued to have problems after the IF run supporting DEIMOS• Problem may be resolved – need to test

– Minor – most in process of being addressed» Facility

• AO offload problem (possible interaction from IF angle tracker)• K2 AO Wavefront controller crashes

» Interferometer• Sequencer crashes• Aligner crate reboots • Telemetry server memory growth – solved

• Goal is to achieve ORR metric performance on next two runs

KISWG 10/29/2003 11

KECKINTERFEROMETER

Performance, functionality, efficiency goals (2/3/2003 pre-ORR presentation)

• These follow from the desire to maximize the science achievable at the specified performance level

KISWG 10/29/2003 12

KECKINTERFEROMETER

Efficiency GoalCurrent performance Plans prior to ORR

Scans 6 /hr 5 /hr, peak

Continued development of telescope sequencer; optimization of IF sequencer

LDL reconfiguration time

10 min, automatic 45 min, manual

Completion of autoalignment system

Camera mode/parameter reconfiguration time 10 min

10 min for clock rate change, only

Need to complete camera upgrades for additional filters

Performance, functionality, efficiency goals, 1 (2/3/2003 pre-ORR presentation)

KISWG 10/29/2003 13

KECKINTERFEROMETER

0

1

2

3

4

5

6

7

8

9

10

Jan-03 Feb-03 Mar-03 Apr-03 May-03 Jun-03 Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04

avg scans /hr

peak scans /pr

ORR goal

ins t uptim e

fac uptim e

ins t req/fac goal

fac req

Inst. uptime req. / fac. uptime goal

Fac. uptime req.

Scans/hr goal

Peak

Avgs

System uptime & scans per hour

KISWG 10/29/2003 14

KECKINTERFEROMETER

Scans per hour

• 86 science plus engineering scans in 22.5 clear hours

• Peak scans on each of 4 nights:– 2, 3, 6, 8

• Sustained high throughput impacted by– System faults– Ease of observing target

» Faint targets, requiring optimization of AO, KAT, FATCAT, or which end up not being trackable, count against scans per hour

» Weather (light to moderate cirrus, which doesn’t count against weather loss)

– May need to qualify our 6 scan/hr goal to apply to sources not pushing system limits

Night 3 of Oct run

Peak scans/hrfor this night

KISWG 10/29/2003 15

KECKINTERFEROMETER

Efficiency GoalCurrent performance Plans prior to ORR Status

Scans 6 /hr 5 /hr, peak

Continued development of telescope sequencer; optimization of IF sequencer 3.8 avg in Oct

LDL reconfiguration time

10 min, automatic 45 min, manual

Completion of autoalignment system

Currently still manual - working toward having this automated for ORR

Camera mode/parameter reconfiguration time 10 min

10 min for clock rate change, only

Need to complete camera upgrades for additional filters OK

Performance, functionality, efficiency goals, 1updates to 2/3/2003 pre-ORR presentation

KISWG 10/29/2003 16

KECKINTERFEROMETER

Instrument requirements goal status plans prior to ORR StatusSpectrometer configurations [y/n] WL: K, Spec: K4 y y available OK

WL: K, Spec: K8 y n upgrade plannedHardware in place, but not planning to provide

WL: H, Spec: H4 y n upgrade planned Demo'd in Oct

Reconfiguration guidelines

LDLbetween sequences

once per night, max

completion of autoaligment system

Currently once per night, max - working on automation for up to 3 moves per night

Camera modes/parametersbetween sequences

once per night, max

mostly requires more experience with mode switching OK

Calibrations provided per star

K1 flux, K2 flux, total flux, background y available OK

Additional nightly calibrations

spectrometer calibration filter flux n initial tests in Nov

OK (equivalent achieved through initial alignment)

Performance, functionality, efficiency goals, 2

KISWG 10/29/2003 17

KECKINTERFEROMETER

Other Target requirements Goal status plans prior to ORR StatusMagnitude R band (AO, autoframerate) 11 10.5

need to test systematically Data taken at R=11; need to assess

Off-axis AO reference allowed [y/n] y n

will be evaluated: could be available for special-purpose observing Not planned

J band (angle tracker @ 100 Hz) 8 9 OK J band (angle tracker @ 20 Hz) 10 ?

need to test systematically OK

Minimum K-band V2 for tracking (bright sources) 0.1

0.1 demonstrated at 500Hz need to test at 200 Hz

OK for trackability - have not assessed data quality at very low V2

Zenith angle2.5 - 50 deg

have tracked to >50 deg OK

Performance, functionality, efficiency goals, 3

KISWG 10/29/2003 18

KECKINTERFEROMETER Procedural Requirements for the ORR

• From the PLRA: • “Provide an operational infrastructure including trained staff,

facilities, hardware and software, so that the Keck Interferometer will operate as a facility for use by the community scientists.”

KISWG 10/29/2003 19

KECKINTERFEROMETER Procedural Requirements for the ORR

• From LaPiana: “Need (A) requirements which you can track against your progress to actually achieve the metric performances, and the (B) ability to do it routinely and having an organization in place that is well trained to do so with robust and stable software (at WMKO & MSC).”

• Questions to address at the ORR from LaPiana:1. Will the ground data system (at WMKO and at MSC)

implementation and testing be complete prior to the start of routine science operations?

2. Will the required operations plans and procedures, including those for anomaly responses, be completed and tested prior to start of routine science operations?

3. Are the operations organization, staff and facilities in place (WMKO and MSC)?

4. Will the necessary training activities be completed prior to start of routine science operations?

5. Are the necessary set of commands and sequences ready to support nominal operations? Will all action items, liens, etc., be satisfactorily dispositioned?

6. Are the risks well understood and acceptable for routine science operations?

KISWG 10/29/2003 20

KECKINTERFEROMETER 1. Ground Data System Completion

• Will the ground data system (at WMKO and at MSC) implementation and testing be complete prior to the start of routine science operations?

• At the pre-ORR: The primary demonstration will be successful CARA operation of the interferometer for V2 science. • We will track this performance over the next ~ 7 runs. • The last 3-runs prior to the ORR will be performed at the full ORR

requirement level.

• CARA personnel have successfully operated the interferometer for most of these runs. We have tracked performance during these runs. Many, but not all, of performance numbers demonstrated in Oct. run.• At most can demonstrate full ORR-level compliance during only 2 runs

(Nov. & Jan.) prior to ORR.

KISWG 10/29/2003 21

KECKINTERFEROMETER 2. Operations Plans & Procedures

• Will the required operations plans and procedures, including those for anomaly responses, be completed and tested prior to start of routine science operations?

• Operations procedures for daytime checkout & night-time operation are in place.

• Anomaly responses need completion• Including spares & troubleshooting training

• Configuration management (software, opto-mechanical & electronics) need to be defined & agreed upon• Especially important during ongoing development

• Major software upgrade will happen with RTC 2.1• Also need upgrade plan (plan to implement upgrades without

compromising operational system).• Would be easier if some subsystems had already been handed over to

CARA• Need to approve/implement plan for maintaining cleanliness of

optics/basement

KISWG 10/29/2003 22

KECKINTERFEROMETER 2. Operations Plans & Procedures

• Current operations procedures:1. Punchlist with new priorities distributed after each run. Reviewed bi-

weekly.2. Schedule for summit activities maintained. Reviewed weekly.

Incorporated in Observatory scheduling.3. Daytime summit activities entered into IFDEV & Observatory daylog.4. Observing support personnel (daytime, nighttime & on-call) list

distributed prior to run.5. Setup & observing procedures available on-line (next slide).6. Afternoon pre-observing checklist/procedure performed.7. Plan for night’s observing distributed.8. Observing log maintained & published to telescope nightlog.9. Post-run summary distributed, including priorities for next run.10. Punchlist with old priorities closed out.11. Post-run review held with Observatory/Project management.

KISWG 10/29/2003 23

KECKINTERFEROMETER

OperationsProcedures

KISWG 10/29/2003 24

KECKINTERFEROMETER 3&4. Operations Organization & Training

• Are the operations organization, staff and facilities in place (WMKO and MSC)?

• Will the necessary training activities be completed prior to start of routine science operations?

• Organization• John Gathright has assumed Operations Coordinator role

• Responsible for coordinating/scheduling/staffing activities in support of observing runs, observing prep & daytime experiments

• 2 (of 3) technicians report directly to him

KISWG 10/29/2003 25

KECKINTERFEROMETER 3&4. Operations Organization & Training

• Facilities• Control rooms set up at headquarters & summit• Control rooms routinely used for interferometer operation• Instrument performance capabilities need to be made available to the

community for observing planningCategory Subsystem Status Tasks prior to ORROptical Path

Adaptive Optics Operational Fix occassional crashesDual Star Module Operational - on-axis beam onlyCoude train Operational - on-axis beam onlyBeam transport optics Operational - on-axis beam onlyLong Delay Lines Operational for remote positioning Automated positioning & alignmentFast Delay Lines Operational - only 1 tracks, both slewKeck Angle Tracker OperationalFringe Tracker Operational More remote control

MetrologyAccelerometer system OperationalCT metrology system Operational Crashes need resolutionTip/tilt metrology system Operational Not used

AutomationTelescope sequencer Operational Needs keywords to select optionsIF sequencer OperationalTelemetry OperationalArchiver OperationalAutoaligner Used as part of manual process Automation under developmentAutofill system Operational EPICS port still needed

KISWG 10/29/2003 26

KECKINTERFEROMETER 3&4. Operations Organization & Training

• Staff• 3 Interferometer specialists trained in operation of Interferometer for

daytime alignments, checkouts & tests, & night-time operations• 2 technicians trained in daytime alignments & checkouts• Plan to hire an interferometer operator & an interferometer specialist to

relieve operational load on above personnel• Software engineers (2) able to support day & night-time troubleshooting. • Electronics engineer able to support daytime troubleshooting. • Mechanical engineer gaining some instrument familiarity.

Category IF Specialists Software EE ME TechsItem JG MH CN WD KS KT AR JB EA IL TS

SystemCoude alignment T T T t T t TBeam train align T T T t T t TPre-observing checklist T T T T t TIF Sequencer operation T T T P P P tTelescope seq operation T TMSC tools operation T T TAO setup & operation t T

KISWG 10/29/2003 27

KECKINTERFEROMETER 3&4. Operations Organization & Training

• Staff• Currently 1 JPL & 1 MSC person participates in each run. Can we fill

their roles after the ORR? • JPL person really helps with troubleshooting

• We are gaining more familiarity with instrument & resolving problems (JPL still on-call)

• MSC person really understands science priorities• We need more insight &/or MSC on-call

KISWG 10/29/2003 28

KECKINTERFEROMETER 5. Commands & Sequences

• Are the necessary set of commands and sequences ready to support nominal operations? Will all action items, liens, etc., be satisfactorily dispositioned?

• Alignment• Procedures exist and are used.• All mirrors and shutters needed are remotely controlled.

• Pre-Observing checklist• Checklist exists and routinely used by CARA personnel.

• Observing procedure/checklist• CARA personnel routinely operate.

• MSC tools procedures• CARA personnel recently operating.

• Post-Observing• Tapes written and shipped.

KISWG 10/29/2003 29

KECKINTERFEROMETER 5. Commands & Sequences

• Target acquisition (Manual & Automated steps)1. Interferometer Sequencer (IS) operator selects target from list

• IS loads parameters & initiates Telescope Sequencer (TS)• IS configures FDLs for observation

2. TS initiates the following (can change for repeat observation):• Telescope control system slews, then tracks, the telescopes• Auto-acquisition tool positions brightest target on acquisition camera

for AO system• AO auto-setting tool optimizes AO parameters on target• AO operator reports AO loops locked to IS operator

3. AO system offloads tip/tilt and focus to the telescope4. IS operator closes Angle Tracker (KAT) loop:

• KAT loop automatically offloads IF tip/tilt mirror to AO wavefront sensor offsets

• AO operator unloads offsets to field steering mirrors if > threshold5. IS operator initiates IS to acquire fringes

• IS scans to peak signal on fringe tracker• IS searches for fringes

KISWG 10/29/2003 30

KECKINTERFEROMETER 5. Commands & Sequences

• Remaining issues:• Automated night-time LDL repositioning (manual & too slow)

• This is currently a high priority• Automated daytime alignment (labor & telescope intensive)

• Likely a lien at the ORR• User interface is complex & distributed (can make mistakes)

• Plan to implement a simpler quick-look• Better version will come after RTC 2.1 upgrade

• Observing sequence automation should be improved to achieve scans/hour with reduced human interaction• Selectable level of telescope sequencer operation• Offloading wavefront sensor centroids to FSMs• Reuse of AO parameters

• Remote configuration• A few remaining hands-on operations

KISWG 10/29/2003 31

KECKINTERFEROMETER 6. Risks

• Are the risks well understood and acceptable for routine science operations?

Risks identified at ORR requirements review:• Weather, wind shake and seeing

• There are nights when IF observing simply cannot be done productively or when only bright targets can be used.

• Best choice in some conditions would be to switch to another instrument program that can succeed with poor seeing.

• Hardware failures• Mitigation: List of critical spares and backup approaches to be identified

& implemented prior to ORR.

KISWG 10/29/2003 32

KECKINTERFEROMETER 6. Risks

• Development during operations• Mitigation: Configuration control.

• Observatory-wide software configuration plan currently being approved. Will apply to Interferometer & AO ops.

• Inteng & tag of intdev nearly in place to protect software.• ECR process in place for AO opto-mechanics.

• Mitigation: Careful coordination of development & operations activities (using current operations procedures).

• Number of shared risk & TAC allocated V2 science nights• Too many nights could result in inadequate CARA involvement in

development &/or inadequate access to interferometer for development.

• Possible mitigation: • Limit the number of science nights.• Additional operations staff.

• New: Load on CARA staff if system requires lots of hand-holding

KISWG 10/29/2003 33

KECKINTERFEROMETER

ORR readiness summary

• Will probably be OK on V2 accuracy spec – need to complete analysis of Oct data

• Uptime still problematic

– Working on major (and minor) problems from Oct

– With satisfactory resolution of major, and several of the minor problems, should be possible to achieve in November & Jan.

• Scans per hour still low, although brief good peak performance

– Some issues with brightness of target – will not apply if target requires extensive optimization

– Improvements to system uptime and user interface should improve performance

• OK on most other areas, except for LDL automation, which is a current focus