csts-file transfer service discussions (2)
DESCRIPTION
CSTS-File Transfer service discussions (2). CNES position. File Content Description Define manifest files identifying File content (to the level that the recipient needs to know) File format? Processing Instructions Combine with content description? Pull Model, Push Model, or both? - PowerPoint PPT PresentationTRANSCRIPT
19/05/2011 CSTS File transfer service discussions
CSTS-File Transfer service discussions (2)
CNES position
19/05/2011 CSTS File transfer service discussions
• File Content Description– Define manifest files identifying
– File content (to the level that the recipient needs to know)– File format?
• Processing Instructions– Combine with content description?
• Pull Model, Push Model, or both?• [Mouy Francois-Xavier] No needs identified yet for push model (but still a little bit of work to do)
• Subscription and automatic push? [Mouy Francois-Xavier] for push model • Subscription and notification of new data? [Mouy Francois-Xavier] how send notifications if
provider not bind. Bind return could notify a list of avaliable data
• File Management Services (see dedicated slide)• File Directory / Catalogue Servcies• File Management [Mouy Francois-Xavier] for push model
– Create / Delete– Move– Local copy
• Directory Management [Mouy Francois-Xavier] for push model – Create / Delete / Copy
• Storage Control [Mouy Francois-Xavier] needs to be configurable – Duration of Storage– Maximum Volume (per client?)
19/05/2011 CSTS File transfer service discussions
• Security (see dedicated slide)• Off the shelf FT protocols support security but must be configured
and depending on features used may need supporting infrastructure• Firewall issues• Access Control
– At what level? (mission, individual accounts, …) [Mouy Francois-Xavier] Mission level seem to be the right level
– Access Control Groups – More complex schemes
• What level of privacy / access control is needed? [Mouy Francois-Xavier] We need to be able to provide this kind of features before security person's asks – E.g. can user X see that data are stored for user Y? [Mouy Francois-
Xavier] he shouldn't ;-) • Key management issues• Others?
19/05/2011 CSTS File transfer service discussions
Reading Martin initial thoughts.– Protocol choices are wide. If we use one or other
protocol, it comes with all this complexity (advantages, drawbacks).
– Some features of standards COTS are not useful in our context, but you have to tune and secure those features anyway.
– Most of the difficulty come with the push model (security, behavior), the pull model is very near the existing offline model.
Initial CNES position:– Firstly, we thought we only need the pull model which
is quite simple than the push model. – Furthermore, File transfer function is already done by
other ways. (ftp servers in parallel)– Therefore, why redo it ?
19/05/2011 CSTS File transfer service discussions
• However, Experience show us that deploying a file transfer feature with common COTS is not quite easy that it is on paper (e.g. with ftp: passive form, active , dynamic port ranges, firewalls, user management, … ).
• Using CSTS could simplify the integration because the network security aspects are already done. A part of the reporting too (notifications).
• CNES think CSTS could be a good « Secured Pipe » to the Ground center.– A binded instance is maintained (Heartbeat)– The Start/Stop mechanisms could be useful to drive
providers behavior– We are able to use CSTS services via a high level of
firewalls system
19/05/2011 CSTS File transfer service discussions
Candidate standards (from initial thoughts)
• File Transfer Standards– FTP, FTPS, SFTP (SSH based FT)– HTTP / HTTPS ?– Others?
• CCSDS Standards– XFDU ?– Others?
19/05/2011 CSTS File transfer service discussions
XFDU (xml formatted data unit)
• GOAL : organizing transferred data in an Open Archive environment (OAIS)
A container (zip file) with
a descripition of its content in a manifest file.
The logical description points to the physical content which can be either local or distant
19/05/2011 CSTS File transfer service discussions
XFDU description
• The Information Package Map contains the logical view of the package. It’s a hierarchical xml tree representing the content of the package. Each leaf of the tree is a Content Unit and can be referred to from the other parts of the package (i.e. information is linked by the Content Unit reference ID).
• The Data Object Section contains all the physical information needed to get the file objects out.
• The Metadata Section records all of the metadata for all items in the package.
• The Behavior Section associates executable code with the content of the package.
19/05/2011 CSTS File transfer service discussions
XFDU manifest structure
The structure map is a hierarchical view of logical references which are pointing to physical data, metadata and specific behaviors.
19/05/2011 CSTS File transfer service discussions
XFDU : benefits
• Unique Identification of the data ( ContentUnit ID)
• Interoperability by using an XML schema description
• Hierarchical structure of the information• Data physical access (DataObject) separated
from the logical concepts (ContentUnit)• Metadata associated with the ContentUnit• Specific processes associated with the
ContentUnit (BehaviorObject)
19/05/2011 CSTS File transfer service discussions
XFDU : drawbacks
XFDU developed for approaching needs but not totally the sames (multi-parts, files, cdrom)
• Few implementations due to standard complexity.:– NASA XFDU library, ESA SAFE’s archive for
ENVISAT data, CNES’s PAIMAS
• Behavior function not really validated
19/05/2011 CSTS File transfer service discussions
CSTS-XFDU : conclusion
• The data organization has been studied earlier by the MOIMS/IPR group. XFDU is an answer to csts concerns with the following conditions: – Needs of an XFDU « light » to answer the
requirements.– Needs to study the behavior statement.
.
19/05/2011 CSTS File transfer service discussions
CSTS-XFDU : conclusion
• XFDU (simplified one) Could be a good system to integrate the file transfer feature.
• The behavior side of XFDU could be a response to File transfer and more …
• CNES thinks it could be a possible cooperation and some people are interested to study those aspects …
19/05/2011 CSTS File transfer service discussions
19/05/2011 CSTS File transfer service discussions
Some use cases at CNES
• Distant management of stations– Management by xfdu of TM archived in station– TM playback via CSTS-offline service
• Inter-facilities scheduler– Distant scripts used by the scheduler managed by
xfdu– Launch via services CSTS– Re-use of CNES well-known scheduler GUI
• …
19/05/2011 CSTS File transfer service discussions
Let’s take an example….
XFDU over CSTS for playing recorded telemetry
19/05/2011 CSTS File transfer service discussions
Telemetry transfert and play
• A facility wants to transfert simulated telemetry to a station and to be able to play it back later.
• The user build an xfdu package (X1) with the TM file and two scripts (play and rewind)
• All the configuration and security concerns to be solved by the management layer.
19/05/2011 CSTS File transfer service discussions
CSTS FT
user
TM 1
PLAY
REW
XFDU X1
CSTS FT
provider
bind
STEP 1
CONNEXION
19/05/2011 CSTS File transfer service discussions
CSTS FT
user
TM 1
PLAY
REW
XFDU X1
TM 1
PLAY
REW
XFDU X1
CSTS FT
provider
bind
START PUT X1
STEP 2
XFDU TRANSFERT
19/05/2011 CSTS File transfer service discussions
CSTS FT
user
TM 1
PLAY
REW
XFDU X1
TM 1
PLAY
REW
XFDU X1
CSTS FT
provider
TM 1
PLAY
REW
bind
START PUT X1
INSTALL
STEP 3
XFDU AUTOMATICINSTALL
19/05/2011 CSTS File transfer service discussions
CSTS FT
user
TM 1
PLAY
REW
XFDU X1
TM 1
PLAY
REW
XFDU X1
CSTS FT
provider
TM 1
PLAY
REW
bind
START PUT X1
START LIST
START LIST X1
PLAY, REW
X1
INSTALL
STEP 4
The user lists all available xfdus
and their behaviors
19/05/2011 CSTS File transfer service discussions
CSTS FT
user
TM 1
PLAY
REW
XFDU X1
TM 1
PLAY
REW
XFDU X1
CSTS FT
provider
TM 1
PLAY
REW
bind
START PUT X1
START LIST
START LIST X1
START RUN X1 PLAY
TM 1
PLAY, REW
X1
INSTALL
STEP 5
The user plays the TM1 telemetry back