srm basic spec lbnl proposal
DESCRIPTION
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N. SRM Basic Spec LBNL Proposal. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/1.jpg)
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N
SRM Basic Spec
LBNL Proposal
Arie Shoshani, Alex Sim, Junmin Gu, Kurt StockingerScientific Data Management Group
Lawrence Berkeley National Laboratory
![Page 2: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/2.jpg)
Summary
• LBNL proposal of SRM Basic specification based on v.1.1• Original v.1.1 methods/arguments in BLUE
Proposed modifications in Green for essential additions with + to the v.1.1 Skyblue for optional additions with ++ to the v.1.1 Red for subtractions from the v.1.1 Purple for renaming and implied updates of the v.1.1
with =>
• Notations:{} … tuple() … list (ordered collection)[] … set (unordered collection)_ … underline as mandatory (as opposed to optional)::= … definition operator| … choice operator (exclusive or)// … commentitalic … recommended
![Page 3: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/3.jpg)
Basically
• From v.2.1.1 or in addition to v.1.1 No space management
o Automatic minimum directory management
o Ls, removeFileso No mkdir, etc.
No permission managemento Somewhat automatic
Some new functionso srmGetSystemInfo // ConfigInfoo Ping
![Page 4: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/4.jpg)
Relationship to v.1.1 and v.2.1.1 (1)
v.1.1 This version v.2.1.1
requestID = integer requestID=string requestToken
=string
requestStatus Separated into RequestSummary and ReturnRequestStatus. FileStatuses from v.1.1 is separated out.
Split into RequestSummary and RequestFileStatus as the response to the request.
FileMetaData Enhanced FileMetaData
MetaDataPathDetail
![Page 5: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/5.jpg)
Relationship to v.1.1 and v.2.1.1 (2)
v.1.1 This version v.2.1.1
RequestFileStatus RequestFileStatus Many different request status. E.g. GetRequestFileStatus, PutRequestFileStatus, etc.
StatusCode Enhanced and
concrete
Explanation added
extensive
automatic storagetype FileStorageType desired to be added
fileStorageType added
![Page 6: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/6.jpg)
Relationship to v.1.1 and v.2.1.1 (3)
v.1.1 This version v.2.1.1
Get Enhanced srmPrepareToGetAdded more parameters
Put Enhanced srmPrepareToPutAdded more parameters
Copy Enhanced srmCopyAdded more parameters
![Page 7: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/7.jpg)
Relationship to v.1.1 and v.2.1.1 (4)
v.1.1 This version v.2.1.1
MkPermanent srmChangeFileStorageType srmChangeFileStorageType
Pin Automatic when file is ready automatic
UnPin srmRelease, or automatic when file lifetime is expired
srmRelease or automatic
Additional pinning to extend lifetime ?
srmExtendedFileLifeTime added
srmExtendedFileLifeTime added
![Page 8: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/8.jpg)
Relationship to v.1.1 and v.2.1.1 (5)
v.1.1 This version v.2.1.1
getRequestStatus srmGetRequestStatus to return RequestSummary
Separated into request type. E.g. srmStatusOf{Get,Put,Copy}Request
Together in getRequestStatus
srmGetRequestFilesStatus added to return detailed information on each file
Separated into request type. E.g. srmStatusOf{Get,Put,Copy}Request
getFileMetadata srmLS with ability to return single directory meta information
srmLS with extensive
Options Enhanced
directory management
![Page 9: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/9.jpg)
Relationship to v.1.1 and v.2.1.1 (6)
v.1.1 This version v.2.1.1
getProtocols srmGetTransferProtocols none
getEstGetTime Absorbed into srmGetRequestStatus
Or srmStatusOf{Get,Put,Copy}Request
Absorbed into srmStatusOfGetRequest
getEstPutTime Absorbed into srmGetRequestStatus
Or srmStatusOf{Get,Put,Copy}Request
Absorbed into
srmStatusOfPutRequest
![Page 10: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/10.jpg)
Relationship to v.1.1 and v.2.1.1 (7)
v.1.1 This version v.2.1.1
setFileStatus srmPutDone Enhanced into srmChangeFileStorageType
Or srmPutDone
AdvisoryDelete srmRemoveFiles ? srmRemoveFiles
Automatic space management
Automatic Space Management
Space Management
Permission Management
minimal Directory management
![Page 11: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/11.jpg)
GET
• In: SURLs, ProtocolsAuthorizationID,
DesiredFileLifeTime,FileStorageType
Out: RequestStatusSummary
RequestID
• Basic assumptions: UserID
• Get()
• In: SURLs, Protocols
Out: RequestStatus
• srmPrepareToGet()
• In:authorizationID,arrayOfGetFileRequest,arrayOfTransferProtocols, userRequestDescription, storageSystemInfo,
totalRetryTime
• Out: requestToken,
returnStatus,
arrayOfFileStatus
![Page 12: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/12.jpg)
RequestStatusSummary
• RequestIDType (Get, Put, … ) State ErrorMessage FileStatusesRetryDeltaTimeSubmitTimeStartTimeFinishTimeEstimatedTimeToStart
• RequestIDRequestTypeStatusCode // RequestErrorCodeStatusExplanation // ErrorDetailsSubmitTimeStartTimeFinishTimeEstimatedTimeToStartTotalFilesInRequestNumberOfQueuedFilesInRequestNumberOfFinishedFilesInRequestNumberOfProcessingFilesInRequest
• requestToken,requestType,
totalFilesInThisRequest,
numOfQueuedRequests,
numOfFinishedRequests,
numOfProgressingRequests,isSuspended
![Page 13: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/13.jpg)
PUT
• Put()
• In: Src_file_names Dest_file_names, Sizes WantPermanent Protocols
• Out: RequestStatus
• In: {Dest_SURLSize }[]
FileStorageType Protocols AuthorizationID OverwriteMode DesiredLifeTime
• Out: RequestStatusSummary RequestID
• srmPrepareToPut()
• In: authorizationIDarrayOfFileRequest,
arrayOfTransferProtocols,
userRequestDescription,
overwriteOption,
storageSystemInfo,
totalRetryTime
• Out: requestToken,
returnStatus,
arrayOfPutFileStatus
![Page 14: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/14.jpg)
COPY
• Copy()
• In: Source_SURLs, Dest_SURLs
• Out: RequestStatus
• In: { Source_SURL,Dest_SURL }[]
AuthorizationID
DesiredLifeTime
FileStorageType
OverwriteModeOut: RequestStatusSummary RequestID
• srmCopy()
• In: authorizationID,
arrayOfCopyFileRequest,
userRequestDescription,
overwriteOption,
removeSourceFilesstorageSystemInfo,
totalRetryTime
• Out: requestToken,
returnStatus,
arrayOfCopyFileStatus
![Page 15: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/15.jpg)
MkPermanent
• MkPermanent• IN: SURLs• OUT: ReturnStatus
• srmChangeFileStorageType()
• In: authorizationID,
arrayOfPath,
desiredStorageType,
spaceToken
• Out: returnStatus,
arrayOfFileStatus
![Page 16: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/16.jpg)
PIN
• Obsolete
• Automatic when file is ready
• In: TURLs
• Out: RequestStatus
![Page 17: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/17.jpg)
UnPin
• UnPin()
• In: RequestIDTURLs
• Out: RequestStatus
• UnPin => Release
• In: RequestIDTURLs | SURLsAuthorizationID
• Out: { TURL | SURL
RequestStatusCode Explanation } []
• srmReleaseFile()
IN: requestToken authorizationID,
siteURLs,
keepSpace
• Out: returnStatus,
arrayOfFileStatus
![Page 18: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/18.jpg)
getRequestStatus
• In: RequestID
• Out: RequestStatus => RequestStatusSummary
• For details: getRequestFilesStatus (previous getRequestStatus which used to include RequestFileStatus and FileMetaData)
• getRequestStatus()
• In: RequestID
• Out: RequestStatus
• srmStatus(Get,Put,Copy)Request
• IN: requestToken,
authorizationID
arrayOfFromSURLs,
• Out: returnStatus,
arrayOfFileSta
![Page 19: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/19.jpg)
getFileMetaData
• getFileMetaData
• In: SURLs
• Out: FileMetaData
• getFileMetaData => LS
• In: SURLsAuthorizationIDIsAllLevelRecursiveListOffsetListCount
infoOnly
• Out: FileMetaData
• srmLs()
• IN: authorizationID,
path,
fileStorageType,
fullDetailedList,
allLevelRecursive,
numOfLevels,
offset,
count
• Out: MetaDataPathDetail[]
returnStatus
![Page 20: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/20.jpg)
FileMetaData
• SURLSizeOwnerGroupPermModeChecksumTypeChecksumValueIsPermanent IsPinned IsCached
• SURLSizeOwnerGroupPermModeChecksumTypeChecksumValueFileStorageTypeStatusCodeStatusCodeLifetimeAssignedLifetimeLeftCreatedTimeLastModifiedTimeFileTypeOriginalSURLSubDirectoryPTR // dirEntries
• path,
status,
size, // 0 if dir
ownerPermission,
userPermission,
groupPermission,
otherPermission
createdAtTime,
lastModificationTime,
owner,
fileStorageType,
fileType, // Directory or FilelifetimeAssigned,
lifetimeLeft,
checkSumType,
checkSumValue,
originalSURL, MetaDataSubPathDetail[]
![Page 21: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/21.jpg)
getProtocols
• In: none
• Out: list of transfer protocols
• None in v.2.1
• Let’s keep this or put into the new srmSystemInfo or srmConfigInfo
![Page 22: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/22.jpg)
getEst[Get/PUT]Time
• Make it Obsolete
• getEstGetTime: • Out- RequestStatus
In- SURLs, Protocols
• getEstPutTime: • Out- RequestStatus
In- Src_file_name, dest_files_names, sizes, wantPermanent,Protocols
• getEstimationTime
• In: Source_SURLDest_SURLSize
AuthorizationID
TransferProtocols
FileStorageType
• Out: RequestEstimation
• None• Similar one is
getStatusRequest() after request is submitted
![Page 23: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/23.jpg)
AdvisoryDelete
• Obsolete
• AdvisoryDelete:
• Out- noneIn- SURLs
• srmRemoveFiles
• In: SURLs authorizationID
• Out: ReturnRequestStatus
• srmRemoveFiles
• IN: requestToken,
authorizationID,
siteURLs
• Out: returnStatus,
arrayOfFileStatus
![Page 24: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/24.jpg)
GetRequestFilesStatus
• In: RequestIDOut: RequestFileStatus []
• In: RequestIDListOffsetListCount
• Out: RequestFileStatus []
• srmStatusOf{Get, Put, Copy}Request
• IN: requestToken,
authorizationID
arrayOfFromSURLs,
• Out: returnStatus,
arrayOfFileStatus
![Page 25: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/25.jpg)
RequestFileStatus
• SURL SizeState FileldTURLOwner, Group, PermModeCheckSumType, CheckSumValueIsPinned, IsPermanent QueueOrderEstSecondsToStart SourceFilenameDestFilenameIsCached
• Source_SURLDest_SURLSizeStatusCode Explanation TURLCheckSumType, CheckSumValueFileStorageTypeEstWaitingTimeOnQueueEstProcessingTimeSourceFilenameDestFilenameRemainingLifetime
• fromSURLInfo,fileSize,
• status,
• estimatedWaitTimeOnQueue,
• estimatedProcessingTime,
• transferURL
• remainingPinTime
• siteURL,
• toSURL,
![Page 26: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/26.jpg)
Ping
• In: none
• Out: alive_response
![Page 27: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/27.jpg)
GetConfig
• In: flag• Out: { value |
list of supported functions}
• e.g. GET, PUT, COPY, DYNAMIC_MKDIR, PUSH_MODE, PULL_MODE,Default_space = 5GB/user, etc.
![Page 28: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/28.jpg)
New types
• FileStorageType VOLATILE, DURABLE, PERMANENT
• FileType FILE, DIRECTORY, LINK
• OverwriteMode NEVER, ALWAYS, WHEN_DIFFERENT
• RequestType GET, PUT, COPY
• StatusCode // RequestStatusCode, FileStatusCode,
RequestErrorCode, FileErrorCode Explanation: human-readable strings // ErrorDetails
• RequestID : from int to string
![Page 29: SRM Basic Spec LBNL Proposal](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813b9c550346895da4d25d/html5/thumbnails/29.jpg)
Conclusion
• Based on SRM v.1.1 and many of v.2.1.1• Included much of Olof’s issues and suggestions to
v.1.1• Operational specification would be bound to Web
Services.