container listing update (liberty swift design summit)
TRANSCRIPT
Copyright©2015 NTT Corp. All Rights Reserved.
Support Last Modified timestamp at Container Listing
Kota Tsuyuzaki [IRC: kota_] [email protected] NTT Software Innovation Center
Copyright(c)2009-2015 NTT CORPORATION. All Rights Reserved.
2 Copyright©2014 NTT Corp. All Rights Reserved.
NTT Confidential
Description: • Currently swift returns "count", "bytes" and "name" attributes for
each container when container listing (with json/xml format) requested. Here, I'd like to discuss to add the last-modified timestamp attribute (maybe put_timestamp value in the container table on account db) to the response as well as object listing (GET container)
Purpose:
• To make retrieving container metadata effective.
Description and Purpose
3 Copyright©2014 NTT Corp. All Rights Reserved.
NTT Confidential
Use Case: • Imagine retrieving the metadata (e.g. x-container-meta-xxx, acl or so on)
PERIODICALLY against to whole existing containers. Currently we have to make a GET account request to find existing containers and then, a HEAD request to each container to get the metadata. Assuming a lot of containers (e.g. >1000) exist in the Swift and almost of containers DOES NOT be updated so frequently, if each request spends 0.1 sec, we have to waste more than 100 sec to retrieve the metadata even though almost of them are same as the one retrieved before .
• If we keep the previous result (something like as a local cache) and there is the last modified timestamp in the GET account response, what we have to do are both f inding updated containers by comparing between the previous result timestamp and each container's last modified timestamp and making HEAD requests for the containers which were updated . (i.e. if only one container updated since the previous result, we just make ONE HEAD request to complete)
UseCase
User
Swift
Collecting Container Metadata and Indexing periodically(e.g. per hour)
Current: HEAD account request x 1 (w/o last-modified) + HEAD container x 1000 = 100.1 sec * Proposal: HEAD account request x 1 (w/ last-modified)+ HEAD (updated) container x n (maybe 1 or 2) = 0.3 sec* *assuming each request 0.1sec
………
Many Containers (e.g. 1000)
4 Copyright©2014 NTT Corp. All Rights Reserved.
NTT Confidential
• https://etherpad.openstack.org/p/liberty-container-listing-update
Discussion Etherpad