lsst dc2 post-mortem february 5–6, 2008 1 dataproperty generally works uses: –fits headers...

LSST DC2 Post-MortemFebruary 5–6, 2008

1

DataProperty

• Generally works• Uses:

– FITS headers– Events– Persistence parameters (additionalData)

1


2

DataProperty boost::any problems

• Must know type before extracting value• Need functions to get an integer/real/string from

anything compatible• Need function to get a dynamically-typed Python

object• Added explicitly typed construction/extraction

functions for Python

2


3

Added and excess functionality

• Added function to combine two DataProperties– For updating WCS parameters

• findUnique() used everywhere– Ability to have multiple identical keys not useful

• SupportFactory overdesigned for DC2– None of functionality used; class itself not used sometimes

3


4

Policy

• Very successful

• Formats– JSON deprecated; should be removed

– XML intended in original design; not necessary?

• File-include functionality not used yet

• Several policy files per pipeline and per run– Policies used only determinable from interpreting other Policies

– Therefore need to persist all for provenance

– Master Policy not currently saved (will be in ticket #315)

• Has get()/getArray() functions for Python

• Does not try to support float, int64_t• Should be merged with DataProperty

– But marked read-only, since modifying Policies is undesirable4


5

Utility Functions

• These functions are in various “utility” modules; they belong in DataProperty/Policy or a common location instead:– fw::getInt64FromAny() — Extract an int64_t value from a boost::any if it holds a signed integral type

– dps.Utils.getDPValue() — Extract a value from a DataProperty into a Python dynamically-typed object

– dps.Utils.dataPropertyToDict() — Convert a DataProperty into a Python dictionary

– ap::extractRequired() — Extract a value from a DataProperty, throwing an exception if not found

– fw::extractPolicyString(), ap::extractPolicyString() — Extract a value from a Policy if the Policy exists, otherwise return a default value

5


6

Policy & DataProperty: design differences

• DataProperty models a single name-value mapping– schizophrenic feature: is it a leaf or a node (or both)?

– children implemented as list• because name is locked to value

• is order important? shouldn't be if names are different

• inefficient access: list searched sequentially

– a single name-value pair is not useful as a modeled object• interface provides no functionality for handling a single pair

• in practice, if you have a lone piece of metadatum, you know what it represents

– unclear semantics

• hierarchy suggests that meaning of named values depends on context

• findAll() suggests absolute meaning of names regardless of context


7

Policy & DataProperty: design differences

• Policy models a set of name-value mappings

– like Java's Properties and C++'s Map

– clean distinction between node & leaf: • Policy=node, other types=leaves

– efficient look-up of name to one or more values

• order is only important for array values

– semantics: meaning is always context-specific

• outer context is assumed by user of Policy


8

Policy capabilities not fully realized

• Automatic loading/importing of policies across multiple files

• Dictionary-based validation

• Request: ability to load default policies from package directory

– policy contents can be loaded from another file (include)

– implemented but not being used

• Request: ability to easily share values at a high level in hierarchy

with lower levels

– designed but not implemented

• PolicyRule

– a way parameterize policy values

– values may change due to changing conditions

lsst dc2 post-mortem february 5–6, 2008 1 dataproperty generally works uses: –fits headers...

Documents