combining semantics an deep learning for intelligent information services

Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017

Combining Semantics and Deep Learning for Intelligent Information ServicesAntrittsvorlesung

Prof. Dr. Harald SackAIFB, 29.11.2017


http://www.telegraph.co.uk/science/2017/10/18/alphago-zero-google-deepmind-supercomputer-learns-3000-years/





The Futile Tries of “Strong” AI20 years of “AI Winter”... https://www.flickr.com/photos/x-ray_delta_one/4128131032

"in from three to eight years we will have a machine with the general intelligence of an average human being", Marvin Minsky (1970)

https://www.flickr.com/photos/x-ray_delta_one/4128131032


Inspired by Biology...

Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20175 https://www.flickr.com/photos/x-ray_delta_one/4128131032 https://commons.wikimedia.org/wiki/File:Blausen_0657_MultipolarNeuron.png

From Biological Neuron to the Artificial Neuron Modell - McCulloch & Pitts (1943)

(Dendrites) (Soma) (Axon)



Cognitive Computing - The MARK 1 Perceptron (1957)

http://techgenix.com/tgwordpress/wp-content/uploads/2017/01/perceptron.jpg

weight update

error


Timeline of Neural Networks

https://www.slideshare.net/deview/251-implementing-deep-learning-using-cu-dnn/4

Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20178 https://www.flickr.com/photos/x-ray_delta_one/4128131032

Deep Convolutional Neural Networks on GPU Supercomputers



Visual Concept Detection

= the ability to learn visual categories in order to automatically identify

new, unseen images of these categories only based on visual content

https://pixabay.com/p-2175353/https://cloud.google.com/vision/

https://pixabay.com/p-2175353/

https://cloud.google.com/vision/


Visual Concept Detection as Machine Learning Task

Supervised Learning:

● Positive images (that depict the concept)

● Negative images (that don’t)

● Classification/Prediction:

○ Test image, if it depicts concept (or not): ??


Size Matters

To achieve high quality results, we need sufficient training data

● Influence of training data size on classification accuracy for

○ Deep Convolutional Neural Networks (CNN)

vs.

○ Aggregated local features and linear predictor (IFV)

● Results:

○ CNNs largely benefit from bigger datasets

○ IFVs are a competitive candidate esp. if only limited

training data is availableC. Hentschel, T. Wiradarma, H. Sack: If we did not have imagenet: Comparison of fisher encodings and convolutional neural networks on limited training data (AVC 2016)


Leveraging Social Media to Improve Visual Content Detection

Extending MIRFLICKR-1M

● 1 Million Flickr images (selection based on interestingness score)

● Additional image metadata (authoritative & user created): text

● How to select appropriate training data?

○ Text: word2vec skip-gram model

to determine related (similar) tags

for a given query

○ Images: visual reranking

to filter images visually similar

to top ranked images

C. Hentschel, H. Sack: Learning from the Uncertain -- Improving Image Classifiers with Community Training Data (i-KNOW 2015)


What Do Classifiers Really See?

● Heatmaps representing the influence of an image region on the classification result

C. Hentschel and H. Sack, What Image Classifiers Really See – Visualizing Bag-of-Visual Words Models (MMM 2015)

Aggregated local features and

linear predictor (IFV)

Deep Convolutional

Neural Networks (CNN)


● Near-human level image classification

● Near-human level speech recognition

● Near-human level handwriting transcription

● Improved machine translation

● Improved text-to-speech conversion

● Digital assistants such as Google Now or Amazon Alexa

● Near-human level autonomous driving

● Superhuman Go playing

What Deep Learning has achieved so far

https://media.wired.com/photos/59268c8ccfe0d93c474309b2/master/pass/GettyImages-627219854.jpg


How to Represent Knowledge?

Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201716 https://upload.wikimedia.org/wikipedia/commons/c/ce/Gottfried_Wilhelm_Leibniz%2C_Bernhard_Christoph_Francke.jpg

The Universal Categories - Aristotle (384–322 BC)

Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201717 https://upload.wikimedia.org/wikipedia/commons/c/ce/Gottfried_Wilhelm_Leibniz%2C_Bernhard_Christoph_Francke.jpg

Calculemus!

Calculus Ratiocinator - Gottfried Wilhelm Leibniz (1646-1716)

„..alle menschlichen Schlussfolgerungen müssten auf irgendeine mit Zeichen arbeitende Rechnungsart zurückgeführt werden, wie es sie in der Algebra und Kombinatorik und mit den Zahlen gibt, wodurch nicht nur mit einer unzweifelhaften Kunst die menschliche Erfindungsgabe gefördert werden könnte, sondern auch viele Streitigkeiten beendet werden könnten, das Sichere vom Unsicheren unterschieden und selbst die Grade der Wahrscheinlichkeiten abgeschätzt werden könnten, da ja der eine der im Disput Streitenden zum anderen sagen könnte: Lasst uns doch nachrechnen!“

Leibniz in a letter to Ph. J. Spener, Juli 1687


Begriffsschrift - Gottlob Frege (1848-1925)

Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201719 https://www.flickr.com/photos/x-ray_delta_one/4128131032

Frames for Represent Knowledge - Marvin Minsky (1974)



The Renaissance of “Soft” AICarol Kaelson/Jeopardy Productions Inc., via Associated Press

From Linked Data to Knowledge Graphs


Knowledge Graphs for Natural Language Processing

rdf:type

dbo:Philosopher

rdfs:subClassOf

dbo:Person

21

Leibniz wrote to Caroline of Ansbach that Newton’s physics was detrimental to natural theology. However, eager to defend the Newtonian view, it was Clarke who responded and the correspondence between both continued until the death of Leibniz.

dbr:Gottfried_Willhelm_Leibniz dbr:Christian_Wolffdbo:doctoralAdvisor

dbo:Philosopher

rdf:type

dbr:Ontology

dbo:notableIdea

text

knowledge base

foaf:name

dbo:birthDate

“Gottfried Wilhelm Leibniz“@de

“1646-07-01”^^xsd:date

“1716-11-14”^^xsd:datedbo:deathDate



22


text

Named EntitiesCommon Entities

Named Entity Linking Language Model● Statistical Context Analysis (co-occurrence)

Knowledge Graph● Graph Analysis (connected components)



23


1. Create potential entity candidates

2. Filter entity candidates by NER type

3. Create induced subgraph of knowledge graph

4. Determine connected components

N. Steinmetz, H. Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions (ESWC 2013)

J. Waitelonis, H. Sack, Named Entity Linking in #Tweets with KEA, (Microposts 2016)


Knowledge Graphs for Question Answering

“Where was Leibniz born?”

Entity Linking wd:Q9047

(Gottfried Wilhelm Leibniz)

Result Type wd:Q618123

(geographical object)

Relation Extraction wdt:P19

(place of birth)

Natural Language


Knowledge Graphs for Question Answering

“Where was Leibniz born?”

Entity Linking

Result Type

Relation Extraction wdt:P19

(place of birth)

wd:Q9047

(Gottfried Wilhelm Leibniz)

Natural Language

wd:Q618123

(geographical object)

SPARQL Query

SELECT ?o WHERE { wd:Q9047 wdt:P19 ?o . ?o wdt:P31/wdt:P279* wd:Q618123 .}


What Knowledge Graphs have achieved so far

● Improved search results on the web

● Answering natural language questions

● Suggest content-based recommendations

● Enable serependitious discoveries

● Enables exploratory search

● Large scale data integration

● Still missing: common sense knowledge


How to combine Deep Learning and Semantics?


▶ Deep Learning for Knowledge Graphs ▶

● NLP and Knowledge Extraction via Deep Learning to populate and extend

Knowledge Graphs

● NLP and Knowledge Extraction via Deep Learning for Ontology Learning to

extend and refine Knowledge Graphs

● NLP and Graph Analysis supported by Deep Learning for Ontology

Alignment and Link Discovery to combine and integrate Knowledge

Graphs

◀ Knowledge Graphs for Deep Learning ? ◀


Word Embeddings (word2vec, glove)


0.2860.792

…−0.177−0.100.109

−0.542…

0.3490.271

● Words are represented as vectors that preserve the linguistic context

● Semantically similar words are represented in close neighborhood within the vector space

● Enable analogies via vector arithmetics

T. Mikolov et al., Efficient Estimation of Word Representations in Vector Space, archivx 2013


Knowledge Graph Embeddings (rdf2vec)

0.2860.792

…−0.177−0.100.109

−0.542…

0.3490.271

● RDF graph are represented as vectors that preserve the semantic context

● Semantically similar entities are represented in close neighborhood within the vector space

● Enable analogies via vector arithmetics

dbr:Gottfried_Wilhelm_Leibniz

dbr:Erhard_Weigel

dbr:Gottlob_Frege

dbr:Hanover

dbr:Leipzig

dbc:German_Mathematician

dbo:academicAdvisor

dbo:influenced

dct:subjectdct:subject

dct:subject

dbr:University_of_Leipzig

dbo:almaMater

dbo:city

dbo:birthPlace

dbo:deathPlace

P. Ristovski et al., RDF2Vec: RDF Graph Embeddings and Their Applications, SWJ 2016

M. Cochez et al, Global RDF Vector Space Embeddings, ISWC 2017


Combined Feature Embeddings for a Compound Knowledge Space

● Various feature vectors

○ Word embeddings

○ Knowledge Graph embeddings

■ Instances■ Ontologies

○ Embeddings for semantically enriched texts

○ Metadata and aggregated features

0.2710.123-0.24

-0.2860.792

…−0.177−0.100.109

−0.542…

0.3490.56

-0.1320.1130.91

...0.560.99

0.2710.123-0.24

-0.334

Text space

Knowledge space

Context space

CompoundKnowledge Space


Towards Neuro-Symbolic Integration

Neuro-Symbolic Systems

1. Translation of symbolic (background)

knowledge into the network

2. Learning of additional knowledge

from examples (and generalisation) by

the network

3. Executing the network

(i.e. reasoning), and

4. Symbolic knowledge extraction

from the network.

Network ensembles

Levels of abstraction

Besold et al.: Neural-Symbolic Learning and Reasoning: A Survey and Interpretation (2017)

specialization


Deep Learning and Semantics for Information Services

Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201735T. Tietz, J. Waitelonis, J. Jäger, H. Sack, refer: a Linked Data based Text Annotation and Recommender System for Wordpress, (ISWC 2016)


From Information Retrieval to Information ExplorationT. Tietz, J. Jäger, J. Waitelonis, H. Sack, Semantic Annotation and Information Visualization for Blogposts with refer, (VOILA 2016)


Prof. Dr. Harald SackFIZ Karlsruhe, Leibniz Institute for Information Infrastructure

AIFB, KIT Karlsruhe

Combining Semantics and Deep Learning for Intelligent Information Services

combining semantics an deep learning for intelligent information services

Education