geometric and semantic matching for cultural heritage...
TRANSCRIPT
![Page 1: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/1.jpg)
Geometric and Semantic Matching for Cultural Heritage
Artefacts
Stephen C. Phillips
IT Innovation
![Page 2: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/2.jpg)
https://www.flickr.com/photos/lunamodule/
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 2
![Page 3: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/3.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 3
![Page 4: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/4.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 4
![Page 5: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/5.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 5
![Page 6: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/6.jpg)
Front Back
Faded
Cracked
Eroded
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
![Page 7: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/7.jpg)
British Museum, London Cyprus Museum, Cyprus
Ashmolean Museum, Oxford Fitzwilliam Museum, Cambridge
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
![Page 8: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/8.jpg)
Jigsaws
in 3D
most pieces missing
all the edges worn off
many puzzles jumbled together
pieces spread across many countries
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
![Page 9: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/9.jpg)
Re-Assembly
Re-Unification
Re-Association _______________
Similarity Search
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 9
![Page 10: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/10.jpg)
Re-Assembly
You have some (eroded) pieces.
The computer fits them together.
This is hard in 2 dimensions, never mind in 3D!
Localised geometry
3D descriptors Localised detail
Colour descriptors
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 10
![Page 11: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/11.jpg)
Re-Unification
Head in the British Museum
Statue in the Cyprus Museum
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 11
Re-unify 3D models of the statue
3D print copies for re-unification
Re-unify statue in a virtual museum
![Page 12: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/12.jpg)
Re-Association
Artefact with semantic description Material, decoration, size, shape, period, style, glaze, …
Data in another museum catalogue
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 12
![Page 13: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/13.jpg)
Re-Association
Find similarities leading to new insights about past cultures
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 13
![Page 14: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/14.jpg)
How?
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 14
![Page 15: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/15.jpg)
What’s the data?
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 15
![Page 16: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/16.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 16
Fragments of Terracotta Statues from Salamis
![Page 17: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/17.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 17
We need to Gather all the Knowledge
Catalogue data with text
descriptions
Archaeological papers
Excavation notes
Chemical analysis
X-ray fluorescence
3D scanning models
![Page 18: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/18.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 18
We need to Gather all the Knowledge
Catalogue data with text
descriptions
Archaeological papers
Excavation notes
Chemical analysis
X-ray fluorescence
3D scanning models
![Page 19: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/19.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 19
We need to Gather all the Knowledge
Catalogue data with text
descriptions
Archaeological papers
Excavation notes
The British Museum has 2.5M objects described in CIDOC CRM Catalogue data from Ashmolean, Fitzwilliam and Cyprus museums is now mapped to the same data model
Object hasNote “…lots of free-form text added by the curator”
Using Natural Language Processing to extract and encode meaning from this text: • References to papers, to catalogue entries • Parts and features • Conservation condition • Measurements • … CIDOC CRM / CRMarchaeo / etc
![Page 20: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/20.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 20
We need to Gather all the Knowledge
Chemical analysis
X-ray fluorescence
3D scanning models
Volume, area thickness,
curvature, …
Colour, texture, distance from convex hull, …
Feature detection
Faceting: front / back
/ fracture
Part annotation
CRMdig CH Artefact Partonomy
All fragments (211) are scanned, in a
variety of resolutions and with a variety of
scanners
![Page 21: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/21.jpg)
Faceting
Assume local fold model
Measure local fold angle
Select locally salient angles
Connect to form facet curves
Determine the facet types
Robust for our fragments
Very few parameters
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 21
90ᵒ
![Page 22: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/22.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 22
Feature Detection
Morphological and stylistic feature detection and characterization
Reasoning on similarity among fragments
−1 −0.5 0 0.5 1
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
x
y
![Page 23: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/23.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 23
Using CH Artefact Partonomy within CIDOC-CRM
![Page 24: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/24.jpg)
How do I use it?
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 24
![Page 25: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/25.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 25
Six views in the dashboard
Inspection
Users can inspect 3D fragments and metadata and run a search
Fragment
3D assets can be processed (faceting, geometric characterisation, feature identification) and annotated
ReAssembly
Exploration
Datasets can be explored according to specific selected properties
History
To preserve the list of operations performed in the session
Clipboard
To save all the data the user is interested in (e.g. notes, models, annotation)
![Page 26: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/26.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 26
Six views in the dashboard
Inspection
Users can inspect 3D fragments and metadata and run a search
Fragment
3D assets can be processed (faceting, geometric characterisation, feature identification) and annotated
ReAssembly
Exploration
Datasets can be explored according to specific selected properties
History
To preserve the list of operations performed in the session
Clipboard
To save all the data the user is interested in (e.g. notes, models, annotation)
![Page 27: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/27.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 27
Six views in the dashboard
Inspection
Users can inspect 3D fragments and metadata and run a search
Fragment
3D assets can be processed (faceting, geometric characterisation, feature identification) and annotated
ReAssembly
Exploration
Datasets can be explored according to specific selected properties
History
To preserve the list of operations performed in the session
Clipboard
To save all the data the user is interested in (e.g. notes, models, annotation)
![Page 28: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/28.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 28
Six views in the dashboard
Inspection
Users can inspect 3D fragments and metadata and run a search
Fragment
3D assets can be processed (faceting, geometric characterisation, feature identification) and annotated
ReAssembly
Exploration
Datasets can be explored according to specific selected properties
History
To preserve the list of operations performed in the session
Clipboard
To save all the data the user is interested in (e.g. notes, models, annotation)
![Page 29: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/29.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 29
Six views in the dashboard
Inspection
Users can inspect 3D fragments and metadata and run a search
Fragment
3D assets can be processed (faceting, geometric characterisation, feature identification) and annotated
ReAssembly
Exploration
Datasets can be explored according to specific selected properties
History
To preserve the list of operations performed in the session
Provenance and argumentation
Clipboard
To save all the data the user is interested in (e.g. notes, models, annotation)
![Page 30: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/30.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
![Page 31: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/31.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 31
Prototype
![Page 32: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/32.jpg)
And Finally: ReAssembly
Use all the data to guide the process
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 32
Selection by
Similarity
Positioning clues
Matching Mating
Jigsaws
in 3D
most pieces missing
all the edges worn off
many puzzles jumbled together
pieces spread across many countries
![Page 33: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/33.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 33
Mannequin
![Page 34: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/34.jpg)
Matching: geometry-based
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 34
![Page 35: Geometric and Semantic Matching for Cultural Heritage ...gravitate-project.eu/sites/default/files/GRAVITATE - Geometric and... · Geometric and Semantic Matching for Cultural Heritage](https://reader034.vdocuments.mx/reader034/viewer/2022042321/5f0abae57e708231d42d0f27/html5/thumbnails/35.jpg)
© Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 35
Mating
Mating digitally mimics gluing of the fragments
We place selected fragments in their optimal relative position based on:
Geometric complementarity
Skin pattern continuity
Semantic constraints
Global alignment
Mathematical morphology used
Final approval of proposed assemblage is requested of the user
(SotA mating from predecessor PRESIOUS)