apecs 2.0 upgrading the apex control...

APECS 2.0 Training, APEX

/ ESO, Jan. '09

1

APECS 2.0APECS 2.0Upgrading the APEX Control SystemUpgrading the APEX Control System

Dirk Muders, Heiko Hafok, MPIfR, BonnDirk Muders, Heiko Hafok, MPIfR, Bonn

APECS History APECS 2.0

Development The Upgrade

Jan. 2009 Changes

compared to APECS 1.1

AP

EC

S =

Ata

cam

a P

athf

inde

r E

xper

imen

t Con

trol

Sys

tem


/ ESO, Jan. '09

2

APECS Origins

ALMA Test Interferometer Control Software (TICS) was immediately usable at APEX due to common hardware interface (CAN bus to Vertex ACU/PTC computers)

Decided to re-use ALMA Common Software (ACS) and TICS and to benefit from large development team (a dozen people at the time in the year 2000)

Hoped for astronomer interfaces, but things developed differently...


/ ESO, Jan. '09

3

TICS

TICS provides CORBA access to the antenna via ACS components to set up observing patterns

These components run under VxWorks to fulfill the real-time CAN bus protocol

Testing the ALMA prototypes was performed mainly via low-level Python scripts using those components directly

A higher-level user interface was planned, but it was not implemented during the test activities


/ ESO, Jan. '09

4

What else was needed for APE(X)CS ?

To use ACS and TICS at APEX all our devices (instruments, auxiliary hardware, etc.) needed to be represented as CORBA components

An astronomer-friendly user interface to set up typical sub-mm observing scans

An online calibration pipeline was required to provide data products for astronomers


/ ESO, Jan. '09

5

APECS Component Interface

In contrast to ALMA, APEX was always supposed to have many different instruments (receivers and spectrometers), facility and PI

Thus we decided to define generic high-level instrument interfaces to be used by all devices of the same kind

This facilitates the setup of observations enormously


/ ESO, Jan. '09

6

SCPI Interface Level

The low-level hardware control systems could not use CORBA directly (ACS footprint was too large; CORBA programming is complex (even in ACS))

Use SCPI (Standard Commands for Programmable Instrumentation) ASCII protocol via sockets to communicate between CORBA components and hardware controllers

SCPI command hierarchy is equivalent to ACS component name space (e.g. APEX:HET345:LO1:MULTI2:biasMode)


/ ESO, Jan. '09

7

SCPI Communication

Decoupling of CORBA and hardware controllers proved to be extremely useful because it isolates APECS from hardware which is developed

by many different groups it allows to plug in simple emulator scripts to

simulate a full system of instruments for APECS developments without real hardware

all CORBA components can be created fully automatically without any further manual interaction

the employed UDP socket protocol allows to debug hardware problems in many different ways

ACS Container

APECS SCPI Setup

Real

Hardware

Simulated

Hardware

(emuEmbSys)

CORBA

Component

ACS

ORB

CORBA

SCPI

SCPI


/ ESO, Jan. '09

9

Multi-Beam FITS (MBFITS)

The lack of a good format to store array receiver data of single-dish radio telescopes led to the development of the MBFITS raw data format

MBFITS stores the instrument and telescope data in a hierarchy of FITS files

MBFITS is now being used at APEX, Effelsberg and the IRAM 30m telescopes


/ ESO, Jan. '09

10

APECS Design

APECS is designed as a pipeline system starting with a scan description (“scan object”) and eventually leading to data products

The pipeline is coordinated by the central “Observing Engine”

Most APECS applications are written in Python, but they use a lot of compiled libraries to speed up computations and transactions

The astronomer interface is an IPython shell


/ ESO, Jan. '09

11

APECS' Astronomer CLI


/ ESO, Jan. '09

12

The APECS Pipeline

For each scan the Observing Engine accepts a Scan Object from a user CLI and Sets up the receivers (tuning; amplifier calibration) Configures the backends Sets up auxiliary devices such as synthesizers, IF

processors or the wobbler Tells the telescope to move in the requested pattern Starts and stops the data acquisition Asks the Calibrator to process the raw data to

produce the final data product or result


/ ESO, Jan. '09

13

The APECS Pipeline (ctd.)

All communications between the Observing Engine and devices or other high-level applications is done via ACS/CORBA


/ ESO, Jan. '09

14

APECS 0.1

The first version of APECS was installed in Chile in September 2003

Much of the time was spent on hardware installations (network, racks, servers)

We had to fight initial problems with failing pressurized hard disk boxes and missing infrastructure


/ ESO, Jan. '09

15

Initial APECS Installation (09/'03)


/ ESO, Jan. '09

16

Initial APECS Installation (09/'03)


/ ESO, Jan. '09

17

APECS 0.2-1.1

Following the initial phase, APECS was debugged and extended a lot during the antenna and early instrument commissioning

We continued to deliver patches and new releases to provide control for more complex instruments and observing modes and to fix bugs

APECS 1.1 is still based on ACS 2.0.1 and runs on machines with RedHat Linux 7.2


/ ESO, Jan. '09

18

The APECS Upgrade

Why upgrade at all ? Mainly the old Linux is making trouble Data rates ever increase and new servers are

needed to handle them RH 7.2 can no longer be installed on them In addition, ACS 2.0.1 has a number of known

issues that have been cured in later versions Old Linux libraries limit the development of new

APECS applications


/ ESO, Jan. '09

19

APECS 2.0 Development

The upgrade plan already came up in 2004 In an effort led by J. Ibsen, ALMA had ported

TICS to ACS 3.0/4.0 The plan was to stay with VxWorks since the

ALMA RT Linux developments were still unstable

The ongoing APEX commissioning delayed our APECS porting

Only in 2007 we found the time to port the APECS and TICS codes to then ACS 5.0


/ ESO, Jan. '09

20

The VxWorks Drama

Unfortunately, we discovered bugs in the ACS property monitoring that required ACS 6.0.4 or higher to be used to avoid container crashes

But that ACS version no longer worked with VxWorks since the ACE/TAO libraries had become to big for the PowerPC architecture

A joint effort of ESO, UTFSM, the Keck Observatory, Remedy IT in the Netherlands and many hours of Bogdan's time reanimated VxWorks under then ACS 7.0.2 again in 2008


/ ESO, Jan. '09

21

APECS 2.0 Software

APECS 2.0 is based on ACS 8.0 and Scientific Linux 5.2 (this is ahead of ALMA who use SL 4.4 !)

Using an up-to-date ACS allows to benefit again from future ALMA developments and bug fixes

Most of the TICS and APECS functionality was kept aligned to APECS 1.1

Some areas are more advanced in APECS 2.0 (e.g. Calibrator → Heiko’s presentation)


/ ESO, Jan. '09

22

APECS 2.0 Software (ctd.)

Basic operator / observer interfaces unchanged Many improvements such as:

Jlog with automatic filter loading New Qt GUI behavior (e.g. Calibrator Client) New Qt widgets available New Python (2.5) New libraries (e.g. SciPy, NumPy)

New Gildas (12/2008) which fixes weighting errors when combining spectra


/ ESO, Jan. '09

23

APECS 2.0 HardwareName Type CPU RAM Disks Location

control2 HP ProLiant DL380 G4

Dual Xeon HT 3.2 GHz

4 GB 2 x High-Altitude SCA Disks (RAID 1)

New server room

instruments2 HP ProLiant DL380 G4



Instrument container

apexdev2 HP ProLiant DL380 G4



New server room

display2 HP ProLiant DL360 G5

Dual QuadCore Xeon E5345

4 GB 1 x High-Altitude SCA Disk, DotHill RAID

New server room

opt2 Industry PC Core2Duo T2500 4 GB 1 x High-Altitude SCA Disk

Instrument container


/ ESO, Jan. '09

24

APECS 2.0 Deployment

Control2

CORBA Services

Observing Engine

Display2

FitsWriter

Calibrator

Instruments2

CORBA Containers with Instr. Comp.

SCPI Parsers

Opt2

Frame Grabber Comp.

DB2 Data Base

Apexdev2

Observing Clients

Monitoring Clients

Instrum

ents

Abm

CORBA Container with Antenna Components

An

tenn a

(AC

U, P

TC

, WIU

)CORB

A

SCPI

CAN

Bus


/ ESO, Jan. '09

25

Software Changes [1]

Names of some variables have changed: APEXROOT → APECSROOT APEXCONFIG → APECSCONFIG APEXSYSLOGS → APECSSYSLOGS

CORBA component name separator “/” instead of “:”, e.g. “APEX/RADIOMETER/RESULTS”

Component access (needs above syntax): apexObsUtils.sc.get_device →

apexObsUtils.sc.getComponentNonSticky


/ ESO, Jan. '09

26


Property access compatible with APECS 1.1: apexObsUtils.getMCPoint with arbitrary separator

chosen from “/”, “:” and “.”, e.g. apexObsUtils.getMCPoint(‘APEX:HET345.state’) or apexObsUtils.getMCPoint(‘APEX.HET345/state’)

List of components: apexObsUtils.sc.COBs_available →

apexObsUtils.sc.availableComponents Enums in MonitorQuery are now strings instead

of integers (e.g. SHUTTER_OPEN)


/ ESO, Jan. '09

27


Web scripts now need to run on “opt2” to fetch data from the DB2 (e.g. for the weather page)

Python wrapper for Gnuplot has been removed from SciPy. Use Pylab / matplotlib instead.

Observer account administration now on “apexdev2”


/ ESO, Jan. '09

28

Operational Changes [1]

Separate AMD maps to access /apexdata in Sequitor (lastarria:/apex_archive) and Chajnantor (display2:/apexdata) to minimize microwave link traffic. Initial reduction on “apexdev2”. Later maybe also on “display2”.

System VNC: control2:1 Observing VNC(s): apexdev2:1 Other VNC(s) (Wobb. GUI, SHFI, etc.):

apexdev2:2/3/4 Observing accounts only on “apexdev2”


/ ESO, Jan. '09

29


Jlog now automatically loads the right filter (“jlog_apecs”); just need to sort by time stamps

Syslogs are now split per hour to reduce link traffic

Syslogs can be loaded into “jlog” for inspection (“jlog –u $APECSSYSLOGS/APECS-<time stamp>.syslog.xml”)


/ ESO, Jan. '09

30


Two new accounts: “apexops”: pointing & focus model, system source &

line catalog administration in $APECSCONFIG “apexdata”: raw & science data and obslog areas

are owned by this account to avoid manipulations via the “apex” account whose password is not secret. The “apexdata” password must not be given to observers !!


/ ESO, Jan. '09

31


As a consequence the data production programs have to run under “apexdata”

This is accomplished by special scripts that use corresponding “ssh” and “sudo” commands: fitsWriter start | stop | restart onlineCalibrator start | stop | restart obsLoggerServer start | stop | restart obsEngine start | stop | restart (for convenience)

Do not use the old restart scripts ! The overall system still runs under “apex” !!


/ ESO, Jan. '09

32


tycho, apexOptRun and MonitorQuery now to be run on “opt2”

DB2 entries now obey the complete naming hierarchy

“stopAllAPECSClients” now stops observer processes too. Always use “restartAPECSServers” which includes stopping the clients. Restart takes only 6 min. now !

ABM console now via “minicom 1” on “opt2”


/ ESO, Jan. '09

33


Offline reductions at the MPIfR are now performed on “display2” (in Bonn) under the “apex” account. The data areas are mounted as /aux/apexa and /aux/apexb.


/ ESO, Jan. '09

34

New Network Setup

In addition to the new APECS hardware, the network has been upgraded to Gbit (reached 55 MB/s (!) for file transfers) and physically split into these subnets at Chajnantor: Control (CORBA, SCPI): 10.0.2.* Data (TCP streams fromt backends to FitsWriter on

“display2-d”): 10.0.5.* (not routed !) Maintenance (everything else: web cams, thin

clients, notebooks, etc.) 10.0.6.* Additional “SciOps” subnet for observational

machines in Sequitor: 10.0.7.* (pular, etc.)

opt 2Optical CameraTerminal to ABM

instruments2instrument control

control2ObsEngine

CORBA-ServicesVNCServer

display2FitsWriterCalibrator

apexdev2Development

APECSvncserver

Hardware devices

Pointing Telescope

Backends

Telescope/WobblerABM

Monitoring Things WEB Cams tcserver

pularVNC client

apexdevVNC client

tacoraVNC client

apexdb2VNC client

VNC Connections

SequitorChajnantor

Control netw

or k, Data netw

or k

Maintenance network

lastarriaArchive

rsync


/ ESO, Jan. '09

36

APECS 2.0 Testing [1]

During the installation a number of tests have been made to verify the performance: Initial drive tests: no strange motions / vibrations Tracking tests at different az/el: like in APECS 1.1 Optical pointing runs: agree with previous results Radio scans with SHFI and SZ (calibration, pointing,

focus, skydip, on, raster, otf (also holo mode), spiral raster) produced MBFITS and Class data as expected.


/ ESO, Jan. '09

37

APECS 2.0 Testing [2]

Throughput tests: AFFTS with 28 x 8192 channels @ 10 Hz (8.75

MB/s !) without delays or other problems SZ beam maps (which failed in Nov. 2008) with 141

subscans, OTF, continuous data (i.e. one 25 minute subscan !) without problems

PBEs and SZ backend are already in the data network. Spectrometers will be changed by B. Klein during his next visit


/ ESO, Jan. '09

38

Issues

APECS 2.0 is good, but not perfect: ABM NC tasks (az/el & wobbler pos.) crashes

sometimes. Being iterated with B. Jeram (ACS). optView GUI freezes after “movie” mode Gildas as of 12/2008 has some bugs. Will need to

update to a more recent version soon. Cron jobs handling syslogs need to be adjusted.


/ ESO, Jan. '09

39

APEX Staff To Do List

All Linux machines in Sequitor should be updated to SL 5.2

At least “llaima” should be available for offline data reduction with SL 5.2 and APECS 2.0 soon

The LDAP and DNS services need to be moved to new servers (ideally off of normal PCs)

The remaining web cams etc. need to be reconfigured to the maintenance subnet

Port “fieldtalk” programs for optical camera to SL 5.2


/ ESO, Jan. '09

40

Conclusions

APECS 2.0 with new network performs much better than the previous system

We are no longer bound to very old hardware and can fulfill requirements of new instruments

Observing modes that started failing last year are now possible again

New observing modes with higher data rates can now be used

Future developments facilitated by new Linux / libraries.

apecs 2.0 upgrading the apex control...

Documents