metadata for material science - at the lightsource bessy ii...bessy ii experiments - general...

Metadata for material scienceat the Lightsource BESSY II

T. Birke1,H. Gorzig1, V. Laux1, T. Mertens1,R. Muller1, M. Ries1, A. Schaelicke1, P.Schnizer1, T. Unold1, L. Vera Ramirez1, J. Viefhaus1

1Helmholtz-Zentrum Berlin (HZB), Germany

11/07/2019

BESSY II

Experiments - general schematic

Some considerations

BESSY II - Accelerator Experiments

BESSY II - beamlines issues

BESSY II - beamlines ongoing work

Summary

Metadata at BESSY II, HZB, 11/06/2019 2

Table of Contents

BESSY II


Some considerations




Summary


Table of Contents

courtesy of M. Ries

Many 6= measurement deviceswith own (meta)data + beamlines

(settings!) to get the light fromthe machine to the experiment.

courtesy of M. Ries

BESSY II


Some considerations




Summary


Table of Contents

Rawdata

Level 3

Meta dataMetadatastore

Process Process Procdata

RawPre-

process Process Pre-procdata

Exp Level 2


Level 3

Meta data

Level 1


no

yes

no

yes

yes

no

Device and settings, notes ...Data Reduction/CleaningBandwidth!!!

Further Data Reduction/CleaningAnalysis / Machine LearningBandwidth AND Storage!!!


Data flow schematic

Rawdata

Level 3



RawPre-


Exp Level 2


Level 3

Meta data

Level 1


no

yes

no

yes

yes

no

Device and settings, notes ...

Data Reduction/CleaningBandwidth!!!



Data flow schematic

Rawdata

Level 3



RawPre-


Exp Level 2


Level 3

Meta data

Level 1


no

yes

no

yes

yes

no

Device and settings, notes ...Data Reduction/CleaningBandwidth!!!



Data flow schematic

BESSY II


Some considerations




Summary


Table of Contents

I What do we consider as metadata?I Auto-generated? → MostI Manual? → Some

I Digital (elog entries)I Analog (handwritten)

I Data lives! → Persistent IDentification ofdata (unique, versioned,...)

I Logbook interface → SometimesI Storage: SQL/noSQL, file formats →

NEXUS + mongodb + ElasticSearchI Scalability: can we handle the future? →

Elastic Stack + NEXUS?I Data Model?I Findability vs amount of metadata

Some side notes (issues acc phys):I Accelerator Physics is

experience based (an Art)(https://www.youtube.com/watch?v=A1L2xODZSI4)

I Trains, cell phones, cars, ...I Running software on different

systemsI Version controlI File formatsI Maintainability of packages/tools

(cookiecutter, code certificates, )I Containerization : Singularity

with a SCIentific File System(SCIF)


Some (meta)data considerations

https://www.youtube.com/watch?v=A1L2xODZSI4

BESSY II


Some considerations




Summary


Table of Contents

I Control System BESSY: Epics 3.14(3.15 under way)

I Epics variables with unique namingconvention (location, device type,etc...) metadata

I all channels logged in archiverI accelerator metadata = dataI Question: which data is metadata

for some expI Meaning meta(data) depends on

settings (changed manually and canNOT be inferred from data)

I We want: Near Real TimeSimulation / Analysis

I Why? Performance optimization,fast recovery, machine protectionand maintenance

I Experiments: beam commissioning


BESSY II - the machine

I PythonI Ophyd for device abstraction (epics, labview, but

also extendable)I Bluesky for experiment control and planningI Nice data model (see right)I Databroker available (base: sqlite and MongoDb)I Suitcase for elasticsearch developed in house

(ohters can be easily produced)I possibility to store data in external files but keep

links to data in database (adaptors can be writtenin straightforward way to load/save the data)

I Generates unique ID for each experimentI Can talk to OlogI METADATA and data hintsI live plotting and fitting


Example: Bluesky and Ophyd


Example bluesky metadata

BESSY II


Some considerations




Summary


Table of Contents

I Different Experiment Control Softwares with different ways to control SampleEnvironment

I Complex software interface protocolsI Time consuming integration of new equipment with little mobility.I No metadata standards for Sample Environments (SECoP with NEXUS?)I Physicists programming drivers.I Human Readable variable names?I No generic work-/dataflow - user/beamline specific, what about

authentication?I New ”workflows” (related to (meta)data) should not change users habits or

create extra overheadI Metadata database for static metadata (see further)I Experiment controls: EPICS, Tango, Sardana → Python?I Common Data Model Architecture vs NoMaD → Data Model?


Beamlines - Let’s complain a bit.

BESSY II


Some considerations




Summary


Table of Contents

courtesy of H. GorzigMetadata at BESSY II, HZB, 11/06/2019 19

courtesy of H. Gorzig

ICAT : metadata catalogue


BESSY II


Some considerations




Summary


Table of Contents

I Many open questions.I Initial work started both from machine side as from experiments side.I EPICS with Bluesky (Python) + Archiver for the machine.I ICAT database for static metadata under development.I NEXUS interfaces for data/metadata under development.I Some limitations can not be solved easily (bandwidth, standardization,

storage, access rights,...)I Working towards FAIR data.


Summary

I Singularity :https://sylabs.io/docs/I Bluesky, Ophyd :https://github.com/blueskyI Scientific Filesytem :https://sci-f.github.io/


Links

https://sylabs.io/docs/

https://github.com/bluesky

https://sci-f.github.io/

metadata for material science - at the lightsource bessy ii...bessy ii experiments - general...

Documents