19/05/2011 CSTS File transfer service discussions

CSTS-File Transfer service discussions (2)

CNES position

19/05/2011 CSTS File transfer service discussions

• File Content Description– Define manifest files identifying

– File content (to the level that the recipient needs to know)– File format?

• Processing Instructions– Combine with content description?

• Pull Model, Push Model, or both?• [Mouy Francois-Xavier] No needs identified yet for push model (but still a little bit of work to do)

• Subscription and automatic push? [Mouy Francois-Xavier] for push model • Subscription and notification of new data? [Mouy Francois-Xavier] how send notifications if

provider not bind. Bind return could notify a list of avaliable data

• File Management Services (see dedicated slide)• File Directory / Catalogue Servcies• File Management [Mouy Francois-Xavier] for push model

– Create / Delete– Move– Local copy

• Directory Management [Mouy Francois-Xavier] for push model – Create / Delete / Copy

• Storage Control [Mouy Francois-Xavier] needs to be configurable – Duration of Storage– Maximum Volume (per client?)

19/05/2011 CSTS File transfer service discussions

• Security (see dedicated slide)• Off the shelf FT protocols support security but must be configured

and depending on features used may need supporting infrastructure• Firewall issues• Access Control

– At what level? (mission, individual accounts, …) [Mouy Francois-Xavier] Mission level seem to be the right level

– Access Control Groups – More complex schemes

• What level of privacy / access control is needed? [Mouy Francois-Xavier] We need to be able to provide this kind of features before security person's asks – E.g. can user X see that data are stored for user Y? [Mouy Francois-

Xavier] he shouldn't ;-) • Key management issues• Others?

19/05/2011 CSTS File transfer service discussions

Reading Martin initial thoughts.– Protocol choices are wide. If we use one or other

protocol, it comes with all this complexity (advantages, drawbacks).

– Some features of standards COTS are not useful in our context, but you have to tune and secure those features anyway.

– Most of the difficulty come with the push model (security, behavior), the pull model is very near the existing offline model.

Initial CNES position:– Firstly, we thought we only need the pull model which

is quite simple than the push model. – Furthermore, File transfer function is already done by

other ways. (ftp servers in parallel)– Therefore, why redo it ?

19/05/2011 CSTS File transfer service discussions

• However, Experience show us that deploying a file transfer feature with common COTS is not quite easy that it is on paper (e.g. with ftp: passive form, active , dynamic port ranges, firewalls, user management, … ).

• Using CSTS could simplify the integration because the network security aspects are already done. A part of the reporting too (notifications).

• CNES think CSTS could be a good « Secured Pipe » to the Ground center.– A binded instance is maintained (Heartbeat)– The Start/Stop mechanisms could be useful to drive

providers behavior– We are able to use CSTS services via a high level of

firewalls system

19/05/2011 CSTS File transfer service discussions

Candidate standards (from initial thoughts)

• File Transfer Standards– FTP, FTPS, SFTP (SSH based FT)– HTTP / HTTPS ?– Others?

• CCSDS Standards– XFDU ?– Others?

19/05/2011 CSTS File transfer service discussions

XFDU (xml formatted data unit)

• GOAL : organizing transferred data in an Open Archive environment (OAIS)

A container (zip file) with

a descripition of its content in a manifest file.

The logical description points to the physical content which can be either local or distant

19/05/2011 CSTS File transfer service discussions

XFDU description

• The Information Package Map contains the logical view of the package. It’s a hierarchical xml tree representing the content of the package. Each leaf of the tree is a Content Unit and can be referred to from the other parts of the package (i.e. information is linked by the Content Unit reference ID).

• The Data Object Section contains all the physical information needed to get the file objects out.

• The Metadata Section records all of the metadata for all items in the package.

• The Behavior Section associates executable code with the content of the package.

19/05/2011 CSTS File transfer service discussions

XFDU manifest structure

The structure map is a hierarchical view of logical references which are pointing to physical data, metadata and specific behaviors.

19/05/2011 CSTS File transfer service discussions

XFDU : benefits

• Unique Identification of the data ( ContentUnit ID)

• Interoperability by using an XML schema description

• Hierarchical structure of the information• Data physical access (DataObject) separated

from the logical concepts (ContentUnit)• Metadata associated with the ContentUnit• Specific processes associated with the

ContentUnit (BehaviorObject)

19/05/2011 CSTS File transfer service discussions

XFDU : drawbacks

XFDU developed for approaching needs but not totally the sames (multi-parts, files, cdrom)

• Few implementations due to standard complexity.:– NASA XFDU library, ESA SAFE’s archive for

ENVISAT data, CNES’s PAIMAS

• Behavior function not really validated

19/05/2011 CSTS File transfer service discussions

CSTS-XFDU : conclusion

• The data organization has been studied earlier by the MOIMS/IPR group. XFDU is an answer to csts concerns with the following conditions: – Needs of an XFDU « light » to answer the

requirements.– Needs to study the behavior statement.

.

19/05/2011 CSTS File transfer service discussions

CSTS-XFDU : conclusion

• XFDU (simplified one) Could be a good system to integrate the file transfer feature.

• The behavior side of XFDU could be a response to File transfer and more …

• CNES thinks it could be a possible cooperation and some people are interested to study those aspects …

19/05/2011 CSTS File transfer service discussions

19/05/2011 CSTS File transfer service discussions

Some use cases at CNES

• Distant management of stations– Management by xfdu of TM archived in station– TM playback via CSTS-offline service

• Inter-facilities scheduler– Distant scripts used by the scheduler managed by

xfdu– Launch via services CSTS– Re-use of CNES well-known scheduler GUI

• …

19/05/2011 CSTS File transfer service discussions

Let’s take an example….

XFDU over CSTS for playing recorded telemetry

19/05/2011 CSTS File transfer service discussions

Telemetry transfert and play

• A facility wants to transfert simulated telemetry to a station and to be able to play it back later.

• The user build an xfdu package (X1) with the TM file and two scripts (play and rewind)

• All the configuration and security concerns to be solved by the management layer.

19/05/2011 CSTS File transfer service discussions

CSTS FT

user

TM 1

PLAY

REW

XFDU X1

CSTS FT

provider

bind

STEP 1

CONNEXION

19/05/2011 CSTS File transfer service discussions

CSTS FT

user

TM 1

PLAY

REW

XFDU X1

TM 1

PLAY

REW

XFDU X1

CSTS FT

provider

bind

START PUT X1

STEP 2

XFDU TRANSFERT

19/05/2011 CSTS File transfer service discussions

CSTS FT

user

TM 1

PLAY

REW

XFDU X1

TM 1

PLAY

REW

XFDU X1

CSTS FT

provider

TM 1

PLAY

REW

bind

START PUT X1

INSTALL

STEP 3

XFDU AUTOMATICINSTALL

19/05/2011 CSTS File transfer service discussions

CSTS FT

user

TM 1

PLAY

REW

XFDU X1

TM 1

PLAY

REW

XFDU X1

CSTS FT

provider

TM 1

PLAY

REW

bind

START PUT X1

START LIST

START LIST X1

PLAY, REW

X1

INSTALL

STEP 4

The user lists all available xfdus

and their behaviors

19/05/2011 CSTS File transfer service discussions

CSTS FT

user

TM 1

PLAY

REW

XFDU X1

TM 1

PLAY

REW

XFDU X1

CSTS FT

provider

TM 1

PLAY

REW

bind

START PUT X1

START LIST

START LIST X1

START RUN X1 PLAY

TM 1

PLAY, REW

X1

INSTALL

STEP 5

The user plays the TM1 telemetry back