ecmwf 1 com intro training course – 2004 ecaccess ecaccess a portal to access ecmwf also known as...

ECMWF 1 Com Intro training course – 2004 ECaccess

ECaccessA portal to access ECMWF

Also known as EasyAccess

Dominique Lucas


Content

ECaccess concepts

telnet, ssh and ftp access

X11 access

ECaccess file management

ECaccess job management

Unattended file transfers

ECaccess vs. ecbatch

Tutorial


ECaccess - Concepts

ECaccess provides a portal to access ECMWF archiving and computing facilities

Strict authentication via SecurID card andX509 certificates

Data integrity/confidentiality guaranteed by SSL


ECaccess - Concepts

ECaccess provides:

File and job management in batch or interactive mode through an extended FTP server

File and job management through a Web browser

A secure telnet access to ECMWF

A secure X Windows access to ECMWF

All this via internet or RMDCN.


ECaccess - Architecture

IN R T ME D R CN NE T

ECaccessMS Gateway Ecgate

ECcert Command

ECaccess Server (Java app)ECaccess

GatewayF I R E W A L L

ECproxy Server (C app)

ECcmd Server

(Java app)

ECios Server (C app)

JSSE-SSL

OpenSSL-SSL

ECMWF Gate- way

FTP

Server

X Server

ECaccess Tools

FTP Client

Telnet Client

Web Browser

MS Work- station

ECaccess CA

(OpenSSL)

Multi tiersECaccess Client, ECaccess Gateway, ECaccess Server, ECaccess ecgate


ECaccess client

Standard Telnet, FTP, ssh and HTTP/S client

Standard X Windows and FTP Server

ECaccess tools (eccert …)



ECaccess gateway

Entry point for all ECaccess users

Verifies ECaccess users authentication(certificates or passcodes)

Implements Telnet, FTP and HTTP/S protocols

Secure tunnels through firewalls to ECaccess Server

Either local gateway installed at your site or ecaccess.ecmwf.int for internet and msaccess.ecmwf.int for RMDCN.



ECaccess Server

Located at ECMWF: ecaccess.ecmwf.int or msaccess.ecmwf.int

Entry point for all ECaccess gateways

Authenticate ECaccess gateways

Provides job and file management functions

Provides functions to monitor file transfers

Provides system access functions

Keeps track of users activity

ECaccess – Architecture


ECaccess ecgate

Runs the ECaccess Certificate Authority (CA)

Provides job management through NQS and LoadLeveler

Provides access to home, scratch and ECFS

Keeps job input, output and error files

Manages spool for unattended file transfers



Use local ECaccess gateways if installed

Internet ECaccess gateways available at:

ctbto.de, dmi.dk, knmi.nl, sma.ch, irmet.ie, smhi.se,

meteo.fr, dwd.de, cerfacs.fr, mercator.ocean.fr,

eumetsat.de, univie.ac.at, met.hu, cscs.ch, inm.es

RMDCN ECaccess Gateways available at:

metoffice.com, inm.es, meteor.gov.tr

If local ECaccess gateway not installed, then use ecaccess.ecmwf.int or msaccess.ecmwf.int

ECaccess – local gateways


ECaccess – telnet/ssh

{mshost}$ telnet ecaccess.ecmwf.int…login: xyzPasscode: XXXXXXWhich system: ecgate…ecgate{~xyz}:1

{mshost}$ ssh [email protected]…login: xyzPasscode: XXXXXXWhich system: ecgate

…ecgate1{~xyz}:1

Where xyz is your ECMWF User ID

On local gateways, port 9023 (telnet) or 9022 (ssh) may be needed.


ECaccess - ftp

{mshost}$ ftp ecaccess.ecmwf.int…login: xyzPasscode: XXXXXX…ftp> ls227 Entering Passive Mode. (193,61,196,110,135,230)150 Opening ASCII mode data connectiondrwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00

ECJOBSdrwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00

ECSCRATCHdrwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00

ECTMPdrwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00

ECFSdrwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00

ECHOME226 Transfer completeftp>

sftp not yet available.

On local gateways, port 9021 may be needed.


ECaccess - ftp

Local environment

ftp entry for batch access

Local file can be dragged and dropped into ecaccessarea.


ECaccess – X11 via telnet

{mshost}$ echo $DISPLAYhostname:0.0{mshost}$ xhost +ecaccess.ecmwf.int ecaccess being added to access control list{mshost}$ telnet ecaccess.ecmwf.int… (Passcode validation required)Which host …: ecgateWhich proxy … : X…ecgate{~xyz}:1 echo $DISPLAY(Something like … )ecaccess.ecmwf.int:<NN>ecgate{~xyz}:2 xcdp&

A control window showing the DISPLAY to be used will appear. This window or an X11 application should remain present to keep the X11 proxy.


ECaccess – ecxterm via ssh

A control window will also appear. Keep it while X11 access required.

{mshost}$ echo $DISPLAYhostname:0.0{mshost}$ ssh –X [email protected] (N.B. Port number IS mandatory)… (Passcode validation required)Which host …: ecgateecgate{~xyz}:1 echo $DISPLAY(Something like)ecaccess.ecmwf.int:<nn>ecgate{~xyz}:2 xcdp&


ECaccess – ecxterm


ECaccess – web – http://ecaccess.ecmwf.int/


ECaccess tools - ECtools

Must be installed at your end on each platform for which access to ECMWF is required.

If you (still) had/have access to “old” ecbatch commands be careful about the $PATH as some commands are “ambiguous” between the two packages e.g. eccert, ecget, ecput.

Help for each command is available with “-help” option:

{mshost}$ eccert –help

ECtools are also available at ecmwf.


ECaccess - eccert Command

{mshost}$ eccert –verboseechost: ecaccess.ecmwf.intecport: 443eccert: /home/xyz/.eccert.crtCertificate requestECMWF user identifier: xyzPasscode from your SecurID card:Certificate saved (855 bytes){mshost}$

Certificate valid for 7 days.

Certificate only needed for file, job and transfer management, NOT required for unattended transfers (see later - ectrans).

Certificate not needed for ECMWF local use of ECtools.


ECaccess environment

Name Purpose Default value

ECDOMAIN Target domain (either ECFS, ECHOME, ECSCRATCH or ECMARS) and optionally target user.

ECHOME

ECDEBUG Enable debug mode False

ECCERT Certificate file location $HOME/.eccert.crt

ECHOST Gateway host name Local host En

viro

nm

ent

Var

iab

les

Oth

er

ECPORT Gateway port number 443


ECaccess File Management

ecls Print a list of files at $ECDOMAIN.

ecdir Same as the previous command but with any system-dependent information.

ecget Retrieve an $ECDOMAIN file and store it on the local machine.

ecreget Same as the previous command, but transfer is continued from the apparent point of failure.

ecput Store a local file at $ECDOMAIN.

ecdelete Delete a file at $ECDOMAIN.

ecmkdir Create a directory at $ECDOMAIN.

ecrmdir Remove an empty directory at $ECDOMAIN.

ecmodtime Get the last modification date of a file at $ECDOMAIN.

ecsize Get the size of a file at $ECDOMAIN.

No support for meta-characters.



{mshost}$ eclsscript.shecaccess-tools.tar.gzecaccess.docecaccess

{mshost}$ ecdir ecaccess10838 drwxr-xr-- 4 xyz systems 96 Mar 14 09:30 .3194 drwxr-xr-x 47 xyz systems 4096 Mar 14

09:30 ..12721 drwxr-x--- 4 xyz systems 96 Mar 13 18:55

client124513 drwxr-x--- 11 xyz systems 2048 Mar 5 11:38

gateway{mshost}$ ecget ecaccess-tools.tar.gz{mshost}$ ecget ecaccess-tools.tar.gz tools.tar.gz{mshost}$ ls *.tar.gzecaccess-tools.tar.gztools.tar.gz{mshost}$



{mshost}$ ecdelete ecaccess-tools.tar.gzDELE command successful{mshost}$ ecput ecaccess-tools.tar.gz{mshost}$ ecmkdir ectestMKD command successful{mshost}$ ecrmdir ectestRMD command successful{mshost}$ ecrmdir ecaccessDirectory not empty{mshost}$



{mshost}$ export ECDOMAIN=ecfs{mshost}$ ecdir10838 drwxr-xr-- 4 xyz systems 96 Mar 14 09:30 .3194 drwxr-xr-x 47 xyz systems 4096 Mar 14

09:30 ..12721 drwxr-x--- 4 xyz systems 96 Mar 13 18:55

backup124513 drwxr-x--- 11 xyz systems 2048 Mar 5 11:38

doc{mshost}$ export ECDOMAIN=“ecfs[zzz]”…{mshost}$ export ECDOMAIN=echome…{mshost}$


ECaccess Job and transfer management

List Ecaccess target queues ecqls

ecjreq Submit a batch request (file is at $ECDOMAIN).

ecjget Get job input, job output and job error.

Submit a batch request (file is local).

Check status of a batch request.

ecjput

ecjls

ecjdel Delete a batch request.

ectreq Initiate a transfer from $ECDOMAIN, using the ectrans spooling mechanism.

ectls List transfers carried out by ectrans.

ectret Retry a transfer.

ectdel Cancel an ectrans transfer (remove it from the spool).

ectinfo Display the target location for a user-id.


ECaccess Job Management

{mshost}$ ecqlsecgate1 NQS submission on ecgate1(INIT=23 ...)hpca LoadLeveler submission on hpca (INIT=21 ...)...{mshost}${mshost}$ ecqls hpcadiag Diagnostic jobs onlytemp Temporary small jobs < 5 Nodesinter_class Default interactive classdebug Default interactive classos Operational serial/single task workns Serial/single task workbench Benchmark classnp Parallel workop Operational parallel work{mshost}$



{mshost}$ ecjreq -helpSyntax: JREQ ECaccess-queue remote-script [args ...] -at - start date (yyyy-MM-dd HH:mm) -nd - no directives within the input script -tg - specify the target gateway name -tr - specifiy the access method (msuser[@destination]) -to - transfer output file when the request ends -te - transfer error file when the request ends -ti - transfer input file when the request ends -tk - keep in spool (default: deleted if transfer

successful) -mu - send mail for the request to the stated address -mb - send mail when the execution/transfer begins -me - send mail when the execution/transfer ends -mf - send mail when the execution/transfer fails



{mshost}$ ecjreq ecgate script.sh –me34850{mshost}$ ecjreq ecgate test.shError opening file{mshost}$

{mshost}$ ecjput ecgate1 test.sh –me34851{mshost}$



{mshost}$ ecjls4421 ecgate WAIT Nov 24

21:333884 [email protected] DONE Nov 18

11:423146 ecgate1 DONE Nov 15

21:33{mshost}$ ecjls 3884

Jobid: 3884 Location: [email protected]

Date/Time: Nov 18 11:42 Status: DONEstdout size: 221stderr size: 219 stdin size: 241Jobid: 4421{mshost}$



{mshost}$ ecjget o34852{mshost}$ ls JOB*JOB-o34852{mshost}$ ecjget o34852 job.out{mshost}$ ls –ail job.out71196 -rw-r----- 1 xyz group 686 Mar 13 19:10

job.out{mshost}$

{mshost}$ ecjdel 34852JDEL command successful{mshost}$

ecjdel kills job running and removes all files.


ECtrans Command

MS Server ms2

MS Workstation

ms3

MS Workstation

ms1

MS Server running the gateway

ECaccess Gateway

(Member State)

Shared File System

MS Network

file: / /

ftp: / /

Internet

ECaccess Server (ECMWF)

$ ectrans –gateway ecaccess.ecmwf.int \ -remote msuser@destination \ –source [location/]filename \ [-target [location/]filename]

destination= ftp://$msuser[name]:$msuser[passwd]@$hostname/$location

ecgate, or hpca:

ECAccess Member State

Gateway:

Unattended file transfers, issued from ECMWF

It is preferable if the ECaccess gateway is installed at your end.

RMDCN


ECtrans - msuser maintenance



MS associations are specific to each ECaccess gateway.


ECtrans Command – at ECMWF

{ec-host}$ ectrans –helpusage: ectrans [-get] -gateway name \

–remote msuser@destination \ -source name [args ...]

-gateway str - target gateway name-source str - source file name-echost str - eccmd host name (default: localhost)-ecport num - eccmd port number (default: 644)-remote str - target user (default: same as ecuser)-target str - target file name (default: same a source)-verbose - verbose mode on-help - this message-get - transfer file from your site to ECMWF-status - check the status on file transfer Default values can also be set by the ECHOST, ECPORT, ECUSER or GATEWAY shell variables.{ec-host}$


ECtrans Command{ecgate1}$ ectrans –gateway ecaccess.ecmwf.int \

-remote from_usl@genericFtp –source ./data.grib \

–target data.grib \ -verboseechost: ecaccess.ecmwf.intecport: 644ecuser: uslsource: ./data.grib target: data.grib keep : falseoption: rejectFile to upload (585 bytes)10129250216245

{ecgate1}$

Users can maintain MS user’s associations on local gateway, via the web interface.

ectrans is available in both directions.


ECaccess transfer Management

{mshost/echost}$ ectreq –helpSyntax: TREQ source [args ...] -remote str - target user (default: current) -gateway str - target gateway name (default:

current) -target str - target file name (default: same as

source) -keep - keep the request in the spool -reject - if the target file already exists

(default) -append - if the target file already exists -resume - if the target file already exists -erase - if the target file already exists{mshost}$ {mshost/echost}$ ectreq job.out10161191420729{mshost}$



{mshost/echost}$ ectret 10161191420729TRET command successful{mshost}$

$SCRATCH used for temporary file storage.



{mshost/echost}$ ectls10161191420729 STOP [email protected] Mar 14

15:1910164593445344 DONE [email protected] Mar 14

15:2210161193432443 DONE [email protected] Mar 14

15:19{mshost}$ ectls 10161191420729

Copyid: 10161191420729MS user: xyz

Hostname: ecaccess.ecmwf.int Access: ECaccess gateway Status: STOP

Error message: No target location for xyz Date/Time: Mar 14 15:19

Source: ./job.out Target: ./job.out

{mshost/echost}$


ECaccess vs. ECbatch

Function ECBATCH software ECaccess software

General Information Not implemented. MS-host:> echelp …MS-host:> ecinfoMS-host:> ectinfo …

Certificate Information MS-host:> eccert –d MS-host:> eccert –expireMS-host:> eccert –ecuser

Certificate generation MS-host:> eccert [-v] MS-host:> eccert

Batch mode Job Submission from MS‑host to ECMWF (script local)

Only supported for Ecgate1, never for HPCF.MS-host:> ecqsub …

Supported for Ecgate1 and for HPCFMS-host:> ecjput …

Batch mode Job Submission from MS‑host to ECMWF (script at ECMWF)

Not implemented. MS-host:> ecjreq …


ECaccess vs. ECbatchFunction ECBATCH software ECaccess software

Return Job’s Stdout/Stderr To MS-host; requires Eccopyd on

MS‑host

To be retrieved from ECMWF

via ecget or ectrans (or via

Web)

Push File on ECMWF‑host to

MS‑host

Requires Eccopyd on MS‑host +

MS‑host:home/.echosts. Ecgate1:>

eccopy …

Requires MS gateway + URL

for user (to be set up by

administrator)

Ecgate1:> ectrans …

File Size Limitations Eccopy limits the file size to 28 MB. File system limits the file size

Push File on MS‑host to

ECMWF

MS-host:> ecput … MS-host:> ecput …

Pull File on MS‑host from

ECMWF

MS-host:> ecget … (No support for

ECFS).

MS-host:> ecget …




Pull File on MS‑host

from ECMWF

MS-host:> ecget … MS-host:> ecget …

Pull File on MS‑host

from ECMWF – restart

after failure

(No support for ECFS). MS-host:> ecreget …

ECMWF File Access

and Management on

MS‑host

Not implemented. ecls, ecdir, ecdelete,

ecmkdir, ecrmdir.

Web File Access and

Management

ecls. Files can be listed,

viewed, stored,

retrieved or deleted.




Web Job Submission

and Management

Not implemented. Job Submission,

termination; status

check; view, store, edit,

retrieve or delete

output.

Telnet to Ecgate1 Not implemented. Provide SecurID code

only once.

X11 to Ecgate1 Not implemented. Provide SecurID code

only once.

Service Discontinued on 31 Dec.

2003

Available since early

2002


ECaccess pointers

http://www.ecmwf.int/services/ecaccess

User’s manual

Administrator’s manual

Registration center (for local ECaccess gateway)

Quick start – check first if local gateway available

http://ecaccess.ecmwf.int/

telnet ecaccess.ecmwf.int

ssh [-X] [email protected]

ftp ecaccess.ecmwf.int

ectrans

ecmwf 1 com intro training course – 2004 ecaccess ecaccess a portal to access ecmwf also known as...

Documents

ecaccess ecaccess ftp

ecaccess ecaccess server

ecaccess ecaccess ecgate

ecaccess ecaccess x11

ecaccess ecaccess ecxterm

ecaccess ecaccess telnetssh

xyz ecaccess

ecaccess area