metadata for material science - at the lightsource bessy ii...bessy ii experiments - general...

Metadata for material scienceat the Lightsource BESSY II

T. Birke1,H. Gorzig1, V. Laux1, T. Mertens1,R. Muller1, M. Ries1, A. Schaelicke1, P.Schnizer1, T. Unold1, L. Vera Ramirez1, J. Viefhaus1

1Helmholtz-Zentrum Berlin (HZB), Germany

11/07/2019

BESSY II

Experiments - general schematic

Some considerations

BESSY II - Accelerator Experiments

BESSY II - beamlines issues

BESSY II - beamlines ongoing work

Summary

Metadata at BESSY II, HZB, 11/06/2019 2

Table of Contents

BESSY II

Experiments - general schematic

Some considerations

BESSY II - Accelerator Experiments

BESSY II - beamlines issues

BESSY II - beamlines ongoing work

Summary

Metadata at BESSY II, HZB, 11/06/2019 3

Table of Contents

courtesy of M. Ries

Many 6= measurement deviceswith own (meta)data + beamlines

(settings!) to get the light fromthe machine to the experiment.

courtesy of M. Ries

Many 6= measurement deviceswith own (meta)data + beamlines

(settings!) to get the light fromthe machine to the experiment.

courtesy of M. Ries

courtesy of M. Ries

courtesy of M. Ries

BESSY II

Experiments - general schematic

Some considerations

BESSY II - Accelerator Experiments

BESSY II - beamlines issues

BESSY II - beamlines ongoing work

Summary

Metadata at BESSY II, HZB, 11/06/2019 8

Table of Contents

Rawdata

Level 3

Meta dataMetadatastore

Process Process Procdata

RawPre-

process Process Pre-procdata

Exp Level 2

Meta dataMetadatastore

Level 3

Meta data

Level 1

Meta dataMetadatastore

no

yes

no

yes

yes

no

Device and settings, notes ...Data Reduction/CleaningBandwidth!!!

Further Data Reduction/CleaningAnalysis / Machine LearningBandwidth AND Storage!!!

Metadata at BESSY II, HZB, 11/06/2019 9

Data flow schematic

Rawdata

Level 3

Meta dataMetadatastore

Process Process Procdata

RawPre-

process Process Pre-procdata

Exp Level 2

Meta dataMetadatastore

Level 3

Meta data

Level 1

Meta dataMetadatastore

no

yes

no

yes

yes

no

Device and settings, notes ...

Data Reduction/CleaningBandwidth!!!

Further Data Reduction/CleaningAnalysis / Machine LearningBandwidth AND Storage!!!

Metadata at BESSY II, HZB, 11/06/2019 9

Data flow schematic

Rawdata

Level 3

Meta dataMetadatastore

Process Process Procdata

RawPre-

process Process Pre-procdata

Exp Level 2

Meta dataMetadatastore

Level 3

Meta data

Level 1

Meta dataMetadatastore

no

yes

no

yes

yes

no

Device and settings, notes ...

Data Reduction/CleaningBandwidth!!!

Further Data Reduction/CleaningAnalysis / Machine LearningBandwidth AND Storage!!!

Metadata at BESSY II, HZB, 11/06/2019 9

Data flow schematic

Rawdata

Level 3

Meta dataMetadatastore

Process Process Procdata

RawPre-

process Process Pre-procdata

Exp Level 2

Meta dataMetadatastore

Level 3

Meta data

Level 1

Meta dataMetadatastore

no

yes

no

yes

yes

no

Device and settings, notes ...Data Reduction/CleaningBandwidth!!!

Further Data Reduction/CleaningAnalysis / Machine LearningBandwidth AND Storage!!!

Metadata at BESSY II, HZB, 11/06/2019 9

Data flow schematic

BESSY II

Experiments - general schematic

Some considerations

BESSY II - Accelerator Experiments

BESSY II - beamlines issues

BESSY II - beamlines ongoing work

Summary

Metadata at BESSY II, HZB, 11/06/2019 10

Table of Contents

I What do we consider as metadata?I Auto-generated? → MostI Manual? → Some

I Digital (elog entries)I Analog (handwritten)

I Data lives! → Persistent IDentification ofdata (unique, versioned,...)

I Logbook interface → SometimesI Storage: SQL/noSQL, file formats →

NEXUS + mongodb + ElasticSearchI Scalability: can we handle the future? →

Elastic Stack + NEXUS?I Data Model?I Findability vs amount of metadata

Some side notes (issues acc phys):I Accelerator Physics is

experience based (an Art)(https://www.youtube.com/watch?v=A1L2xODZSI4)

I Trains, cell phones, cars, ...I Running software on different

systemsI Version controlI File formatsI Maintainability of packages/tools

(cookiecutter, code certificates, )I Containerization : Singularity

with a SCIentific File System(SCIF)

Metadata at BESSY II, HZB, 11/06/2019 11

Some (meta)data considerations

BESSY II

Experiments - general schematic

Some considerations

BESSY II - Accelerator Experiments

BESSY II - beamlines issues

BESSY II - beamlines ongoing work

Summary

Metadata at BESSY II, HZB, 11/06/2019 12

Table of Contents

I Control System BESSY: Epics 3.14(3.15 under way)

I Epics variables with unique namingconvention (location, device type,etc...) metadata

I all channels logged in archiverI accelerator metadata = dataI Question: which data is metadata

for some expI Meaning meta(data) depends on

settings (changed manually and canNOT be inferred from data)

I We want: Near Real TimeSimulation / Analysis

I Why? Performance optimization,fast recovery, machine protectionand maintenance

I Experiments: beam commissioning

Metadata at BESSY II, HZB, 11/06/2019 13

BESSY II - the machine

I PythonI Ophyd for device abstraction (epics, labview, but

also extendable)I Bluesky for experiment control and planningI Nice data model (see right)I Databroker available (base: sqlite and MongoDb)I Suitcase for elasticsearch developed in house

(ohters can be easily produced)I possibility to store data in external files but keep

links to data in database (adaptors can be writtenin straightforward way to load/save the data)

I Generates unique ID for each experimentI Can talk to OlogI METADATA and data hintsI live plotting and fitting

Metadata at BESSY II, HZB, 11/06/2019 14

Example: Bluesky and Ophyd

Metadata at BESSY II, HZB, 11/06/2019 15

Example bluesky metadata

BESSY II

Experiments - general schematic

Some considerations

BESSY II - Accelerator Experiments

BESSY II - beamlines issues

BESSY II - beamlines ongoing work

Summary

Metadata at BESSY II, HZB, 11/06/2019 16

Table of Contents

I Different Experiment Control Softwares with different ways to control SampleEnvironment

I Complex software interface protocolsI Time consuming integration of new equipment with little mobility.I No metadata standards for Sample Environments (SECoP with NEXUS?)I Physicists programming drivers.I Human Readable variable names?I No generic work-/dataflow - user/beamline specific, what about

authentication?I New ”workflows” (related to (meta)data) should not change users habits or

create extra overheadI Metadata database for static metadata (see further)I Experiment controls: EPICS, Tango, Sardana → Python?I Common Data Model Architecture vs NoMaD → Data Model?

Metadata at BESSY II, HZB, 11/06/2019 17

Beamlines - Let’s complain a bit.

BESSY II

Experiments - general schematic

Some considerations

BESSY II - Accelerator Experiments

BESSY II - beamlines issues

BESSY II - beamlines ongoing work

Summary

Metadata at BESSY II, HZB, 11/06/2019 18

Table of Contents

courtesy of H. GorzigMetadata at BESSY II, HZB, 11/06/2019 19

courtesy of H. Gorzig

ICAT : metadata catalogue

Metadata at BESSY II, HZB, 11/06/2019 20

courtesy of H. GorzigMetadata at BESSY II, HZB, 11/06/2019 21

courtesy of H. GorzigMetadata at BESSY II, HZB, 11/06/2019 22

courtesy of H. GorzigMetadata at BESSY II, HZB, 11/06/2019 23

BESSY II

Experiments - general schematic

Some considerations

BESSY II - Accelerator Experiments

BESSY II - beamlines issues

BESSY II - beamlines ongoing work

Summary

Metadata at BESSY II, HZB, 11/06/2019 24

Table of Contents

I Many open questions.I Initial work started both from machine side as from experiments side.I EPICS with Bluesky (Python) + Archiver for the machine.I ICAT database for static metadata under development.I NEXUS interfaces for data/metadata under development.I Some limitations can not be solved easily (bandwidth, standardization,

storage, access rights,...)I Working towards FAIR data.

Metadata at BESSY II, HZB, 11/06/2019 25

Summary

I Singularity :https://sylabs.io/docs/I Bluesky, Ophyd :https://github.com/blueskyI Scientific Filesytem :https://sci-f.github.io/

Metadata at BESSY II, HZB, 11/06/2019 26

Links