![Page 1: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/1.jpg)
CS224W: Machine Learning with GraphsJure Leskovec, Hongyu Ren, Stanford University
http://cs224w.stanford.edu
![Page 2: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/2.jpg)
1. Introduction to Knowledge Graphs
2. Knowledge Graph completion
3. Path Queries
4. Conjunctive Queries
5. Query2Box: Reasoning with Box Embeddings
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 2
![Page 3: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/3.jpg)
¡ Knowledge in graph form§ Capture entities, types, and relationships
¡ Nodes are entities¡ Nodes are labeled with their types¡ Edges between two nodescapture relationships between entities
11/21/19 3Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu
![Page 4: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/4.jpg)
¡ Node types: paper, title, author, conference, year
¡ Relation types: pubWhere, pubYear, hasTitle, hasAuthor, cite
11/21/19 4Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu
![Page 5: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/5.jpg)
¡ Node types: account, song, post, food, channel¡ Relation types: friend, like, cook, watch, listen
11/21/19 5Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu
![Page 6: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/6.jpg)
11/21/19 6Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu
paintedBy
![Page 7: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/7.jpg)
¡ Google Knowledge Graph ¡ Amazon Product Graph¡ Facebook Graph API ¡ IBM Watson ¡ Microsoft Satori ¡ Project Hanover/Literome¡ LinkedIn Knowledge Graph ¡ Yandex Object Answer
11/21/19 7Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu
![Page 8: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/8.jpg)
¡ Serving information
11/21/19 8Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu
![Page 9: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/9.jpg)
¡ Question answering and conversation agents
11/21/19 9Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu
![Page 10: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/10.jpg)
1. Introduction to Knowledge Graphs
2. Knowledge Graph completion
3. Path Queries
4. Conjunctive Queries
5. Query2Box: Reasoning with Box Embeddings
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 10
![Page 11: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/11.jpg)
¡ Publicly available KGs:§ FreeBase, Wikidata, Dbpedia, YAGO, NELL, etc.
¡ Common characteristics:§ Massive: millions of nodes and edges§ Incomplete: many true edges are missing
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 11
Given a massive KG, enumerating all the
possible facts is intractable!
Can we predict plausible BUT missing links?
![Page 12: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/12.jpg)
¡ Freebase§ ~50 million entities§ ~38K relation types§ ~3 billion facts/triples
¡ FB15k/FB15k-237§ A complete subset of Freebase, used by
researchers to learn KG models
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 12
93.8% of persons from Freebase have no place of birth and 78.5% have no nationality!
[1] Paulheim, Heiko. "Knowledge graph refinement: A survey of approaches and evaluation methods." Semantic web 8.3 (2017): 489-508.[2] Min, Bonan, et al. "Distant supervision for relation extraction with an incomplete knowledge base." Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2013.
![Page 13: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/13.jpg)
¡ Given an enormous KG, can we complete the KG / predict missing relations?§ links + type
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 13
missing relation: genre
![Page 14: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/14.jpg)
¡ Edges in KG are represented as triples (ℎ, 𝑟, 𝑡)§ head (ℎ) has relation 𝑟 with tail (𝑡).
¡ Key Idea: § Model entities and relations in the
embedding/vector space ℝ( . § Given a true triple (ℎ, 𝑟, 𝑡), the goal is that the
embedding of (ℎ, 𝑟) should be close to the embedding of 𝑡.§ How to embed ℎ, 𝑟 ?§ How to define closeness?
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 14
![Page 15: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/15.jpg)
¡ Symmetric Relations:𝑟 ℎ, 𝑡 ⇒ 𝑟 𝑡, ℎ ∀ℎ, 𝑡
§ Example: Family, Roommate¡ Composition Relations:
𝑟+ 𝑥, 𝑦 ∧ 𝑟/ 𝑦, 𝑧 ⇒ 𝑟1 𝑥, 𝑧 ∀𝑥, 𝑦, 𝑧§ Example: My mother’s husband is my father.
¡ 1-to-N, N-to-1 relations:𝑟 ℎ, 𝑡+ , 𝑟 ℎ, 𝑡/ , … , 𝑟(ℎ, 𝑡3) are all True.
§ Example: 𝑟 is “StudentsOf”
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 15
![Page 16: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/16.jpg)
¡ Translation Intuition: For a triple (ℎ, 𝑟, 𝑡), 𝐡, 𝐫, 𝐭 ∈ ℝ(,
𝐡 + 𝐫 = 𝐭
Score function: 𝑓; ℎ, 𝑡 = ||ℎ + 𝑟 − 𝑡||
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 16
𝐡 𝐭
𝐫 ObamaNationality
American
Bordes, Antoine, et al. "Translating embeddings for modeling multi-relational data." Advances in neural information processing systems. 2013.
NOTATION: embedding vectors will appear inboldface
![Page 17: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/17.jpg)
¡ Translation Intuition: for a triple (ℎ, 𝑟, 𝑡), 𝐡 + 𝐫 = 𝐭
Max margin loss:
ℒ = ?(@,;,A)∈B,(@,;,AC)∉B
𝛾 + 𝑓;(ℎ, 𝑡) − 𝑓;(ℎ, 𝑡F) G
where 𝛾 is the margin, i.e., the smallest distance tolerated by the model between a valid triple and a corrupted one.
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 17
Valid triple Corrupted triple
NOTE: check lecture 7 for a more in-depth discussionof TransE!
![Page 18: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/18.jpg)
¡ Who has won the Turing award?
¡ Who is a Canadian citizen?
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 18
Win
HintonBengio
Pearl
TuringAward
Canada
Trudeau Bieber
𝐪
Answers!
HintonBengio
Pearl
TuringAward
CanadaCitizen
Trudeau Bieber
Answers!
𝐪
![Page 19: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/19.jpg)
¡ Composition Relations:𝑟+ 𝑥, 𝑦 ∧ 𝑟/ 𝑦, 𝑧 ⇒ 𝑟1 𝑥, 𝑧 ∀𝑥, 𝑦, 𝑧
¡ Example: My mother’s husband is my father.¡ In TransE:
𝑟1 = 𝑟+ + 𝑟/ ü
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 19
𝐱𝐫+ 𝐫/
𝐫1
𝐲𝐳
![Page 20: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/20.jpg)
¡ Symmetric Relations:𝑟 ℎ, 𝑡 ⇒ 𝑟 𝑡, ℎ ∀ℎ, 𝑡
¡ Example: Family, Roommate¡ In TransE:
𝑟 = 0, ℎ = 𝑡 û
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 20
𝐡 𝐭
𝐫
If we want TransE to handle symmetric relations 𝑟, for all ℎ, 𝑡 that satisfy 𝑟(ℎ, 𝑡), 𝑟(𝑡, ℎ) is also True, which means ‖‖
ℎ + 𝑟 −𝑡 = 0 and 𝑡 + 𝑟 − ℎ = 0. Then 𝑟 = 0 and ℎ = 𝑡, however ℎ and 𝑡 are two different entities and should be mapped to different locations.
![Page 21: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/21.jpg)
¡ 1-to-N, N-to-1, N-to-N relations.¡ Example: (ℎ, 𝑟, 𝑡+) and (ℎ, 𝑟, 𝑡/) both exist in
the knowledge graph, e.g., 𝑟 is “StudentsOf”
With TransE, 𝑡+ and 𝑡/ will map to the same vector, although they are different entities.
¡ 𝐭+ = 𝐡 + 𝐫 = 𝐭/¡ 𝐭+ ≠ 𝐭/
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 21
𝐡
𝐭+𝐭/ 𝐫
𝐫contradictory!
![Page 22: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/22.jpg)
¡ TransR: model entities as vectors in the entity space ℝ( and model each relation as vector 𝒓 in relation space ℝP with 𝐌; ∈ ℝP×( as the projection matrix.
¡ ℎS = 𝑀;ℎ, 𝑡S = 𝑀;𝑡¡ 𝑓; ℎ, 𝑡 = ||ℎS + 𝑟 − 𝑡S||
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 22
𝐡𝐡S 𝐭S
𝐭
𝐫
Lin, Yankai, et al. "Learning entity and relation embeddings for knowledge graph completion." AAAI. 2015.
![Page 23: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/23.jpg)
¡ Symmetric Relations:𝑟 ℎ, 𝑡 ⇒ 𝑟 𝑡, ℎ ∀ℎ, 𝑡
¡ Example: Family, Roommate
𝑟 = 0, ℎS = 𝑀;ℎ = 𝑀;𝑡 = 𝑡Sü
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 23
𝐡 𝐭S, ℎS
𝐭
𝑴;
For TransR, we can map ℎ and 𝑡 to the same location on the space of relation 𝑟.
![Page 24: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/24.jpg)
¡ 1-to-N, N-to-1, N-to-N relations¡ Example: If (ℎ, 𝑟, 𝑡+) and (ℎ, 𝑟, 𝑡/) exist in the
knowledge graph.
We can learn 𝑀; so that 𝑡S = 𝑀;𝑡+ = 𝑀;𝑡/, note that 𝑡+ does not need to be equal to 𝑡/!
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 24
𝐡𝐡S 𝐭S
𝐭+
𝐭/𝐫
![Page 25: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/25.jpg)
¡ Composition Relations:𝑟+ 𝑥, 𝑦 ∧ 𝑟/ 𝑦, 𝑧 ⇒ 𝑟1 𝑥, 𝑧 ∀𝑥, 𝑦, 𝑧
¡ Example: My mother’s husband is my father.
Each relation has different space.It is not naturally compositional for multiple relations! û
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 25
![Page 26: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/26.jpg)
Embedding Entity Relation 𝒇𝒓(𝒉, 𝒕)TransE ℎ, 𝑡 ∈ ℝ( 𝑟 ∈ ℝ( ||ℎ + 𝑟 − 𝑡||TransR ℎ, 𝑡 ∈ ℝ( 𝑟 ∈ ℝP,𝑀; ∈ ℝP×( ||𝑀;ℎ + 𝑟 −𝑀;𝑡||
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 26
Embedding Symmetry Composition One-to-many
TransE û ü û
TransR ü û ü
![Page 27: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/27.jpg)
1. Introduction to Knowledge Graphs
2. Knowledge Graph completion
3. Path Queries
4. Conjunctive Queries
5. Query2Box: Reasoning with Box Embeddings
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 27
![Page 28: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/28.jpg)
¡ Can we do multi-hop reasoning, i.e., answer complex queries efficiently on an incomplete, massive KG?
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 28
Query Types Examples
One-hop Queries Where did Hinton graduate?
Path Queries Where did Turing Award winners graduate?
Conjunctive Queries Where did Canadians with Turing Award graduate?
EPFO Queries Where did Canadians with Turing Award or Nobel graduate?
![Page 29: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/29.jpg)
¡ We can formulate link prediction problems as answering one-hop queries.
¡ Link prediction: Is link (ℎ, 𝑟, 𝑡) True?
¡ One-hop query: Is 𝑡 an answer to query (ℎ, 𝑟)?
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 29
![Page 30: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/30.jpg)
¡ Generalize one-hop queries to path queries by adding more relations on the path.
¡ Path queries can be represented by𝑞 = 𝑣\, 𝑟+, … , 𝑟3
𝑣\ is a constant node, answers are denoted by 𝑞 .
Computation graph of 𝑞:
Computation graph of path queries is a chain.11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 30
𝑉?𝑣\
𝑟+ 𝑟/ 𝑟3…
![Page 31: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/31.jpg)
“Where did Turing Award winners graduate?”¡ 𝑣\ is “Turing Award”.¡ 𝑟+, 𝑟/ is (“win”, “graduate”).
Given a KG, how to answer the query?
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 31
Win Graduate
𝑉 𝑉?Turing Award
![Page 32: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/32.jpg)
¡ Answer path queries by traversing the KG.“Where did Turing Award winners graduate?”
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 32
Turing Award
The anchor node is Turing Award.
![Page 33: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/33.jpg)
¡ Answer path queries by traversing the KG.“Where did Turing Award winners graduate?”
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 33
Turing Award
Win
Pearl
Hinton
Bengio
Start from the anchor node “Turing Award” and traverse the KG by the relation “Win”, we reach entities {“Pearl”, “Hinton”, “Bengio”}.
![Page 34: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/34.jpg)
¡ Answer path queries by traversing the KG.“Where did Turing Award winners graduate?”
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 34
Turing Award
Win
Pearl
Graduate EdinburghHinton
McGill
Bengio
Cambridge
NYU
Answers!Start from nodes {“Pearl”, “Hinton”, “Bengio”} and traverse the KG by the relation “Graduate”, we reach entities {“NYU”, “Edinburgh”, “Cambridge”, “McGill”}. These are the answers to the query!
![Page 35: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/35.jpg)
¡ Answer path queries by traversing the KG.“Where did Turing Award winners graduate?”
What if KG is incomplete?
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 35
Turing Award
Win
Pearl
Graduate EdinburghHinton
McGill
Bengio
Cambridge
NYU
Answers!
![Page 36: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/36.jpg)
¡ Can we first do link prediction and then traverse the completed (probabilistic) KG?
¡ No! The completed KG is a dense graph!¡ Time complexity of traversing a dense KG with 𝑉 entities to answer (𝑣\, 𝑟+, … , 𝑟3) of length 𝑛 is 𝒪 𝑉 3 .
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 36
𝑟/
𝑣\
𝑟+𝑣+
𝑣/
𝑣|b|
…
𝑣/
𝑣|b|
…
𝑣+
…
…
![Page 37: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/37.jpg)
¡ Key idea: embed queries!§ Generalize TransE to multi-hop reasoning.
Given a path query 𝑞 = 𝑣\, 𝑟+, … , 𝑟3 ,
𝐪 = 𝐯\ + 𝐫+ + ⋯+ 𝐫3¡ Is 𝑣 an answer to 𝑞? § Do a nearest neighbor search for all 𝑣 based on 𝑓e 𝑣 = ||𝐪 − 𝐯||, time complexity is 𝒪(𝑉).
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 37
𝐯𝒂𝐪
𝐫+ 𝐫/𝐫3…
Guu, Kelvin, John Miller, and Percy Liang. "Traversing knowledge graphs in vector space." arXiv preprint arXiv:1506.01094 (2015).
![Page 38: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/38.jpg)
¡ Embed path queries in vector space.“Where did Turing Award winners graduate?”Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 38
TuringAward
Computation Graph
TuringAward
Embedding Space
![Page 39: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/39.jpg)
¡ Embed path queries in vector space.“Where did Turing Award winners graduate?”Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 39
TuringAward Projection
Computation Graph
Win
Bengio
Pearl
TuringAward
Embedding Space
Hinton
![Page 40: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/40.jpg)
¡ Embed path queries in vector space.“Where did Turing Award winners graduate?”Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 40
TuringAward Projection Projection
Computation Graph
Win
NYU
HintonBengio
Pearl
GraduateMcGill Edinburgh
Cambridge
TuringAward
Embedding Process𝐪
Answers!
![Page 41: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/41.jpg)
1. Introduction to Knowledge Graphs
2. Link Prediction
3. Path Queries
4. Conjunctive Queries
5. Query2Box: Reasoning with Box Embeddings
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 41
![Page 42: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/42.jpg)
¡ Can we answer more complex queries?¡ What if we start from multiple anchor nodes?“Where did Canadian citizens with Turing Award graduate?”
Computation graph of 𝑞:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 42
TuringAward
Canada
Projection
ProjectionProjection
Intersection
Intersection
![Page 43: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/43.jpg)
¡ Can we answer even more complex queries?“Where did Canadian citizens with Turing Award graduate?”
Two anchor nodes: Canada and Turing Award.
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 43
Turing Award
Win
Pearl
Hinton
Bengio
Start from the first anchor node “Turing Award”, and traverse by relation “Win”, we reach {“Pearl”, “Hinton”, “Bengio”} .
![Page 44: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/44.jpg)
¡ Can we answer even more complex queries?“Where did Canadian citizens with Turing Award graduate?”
Two anchor nodes: Canada and Turing Award.
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 44
Turing Award
Win
Canada
Citizen
Pearl
Hinton
Bieber
Bengio
Trudeau
Start from the second anchor node “Canada”, and traverse by relation “citizen”, we reach { “Hinton”, “Bengio”, “Bieber”, “Trudeau”}
![Page 45: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/45.jpg)
¡ Can we answer even more complex queries?“Where did Canadian citizens with Turing Award graduate?”
Two anchor nodes: Canada and Turing Award.
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 45
Turing Award
Win
Canada
Citizen
Pearl
Hinton
Bieber
Bengio
Trudeau
Then, we take intersection of the two sets and achieve {‘Hinton’, ‘Bengio’}
![Page 46: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/46.jpg)
¡ Can we answer even more complex queries?“Where did Canadian citizens with Turing Award graduate?”
Two anchor nodes: Canada and Turing Award.
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 46
Turing Award
Win
Canada
Citizen
Pearl
Graduate
EdinburghHinton
McGill
Bieber
Bengio
Trudeau
Cambridge
We do another traverse and arrive at the answers!
![Page 47: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/47.jpg)
¡ Key Idea: embed queries in vector space“Where did Canadian citizens with Turing Award graduate?”
Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 47
Computation Graph Embedding Space
TuringAward
Canada
Projection
Projection
Win
HintonBengio
Pearl
TuringAward
CanadaCitizen
Trudeau
Bieber
𝐪+
𝐪/
![Page 48: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/48.jpg)
¡ Key Idea: embed queries in vector space“Where did Canadian citizens with Turing Award graduate?”
Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 48
Computation Graph Embedding Process
Win
HintonBengio
Pearl
TuringAward
CanadaCitizen
Trudeau
Bieber
TuringAward
Canada
Projection
Projection
Intersection
Intersection
?𝐪+
𝐪/
![Page 49: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/49.jpg)
¡ How do we take intersection of several vectors in the embedding space?
¡ Design a neural intersection operator ℐ§ Input: current query embeddings 𝐪+, … , 𝐪h§ Output: intersection query embedding 𝐪§ ℐ should be permutation invariant:
ℐ 𝐪+, … , 𝐪h = ℐ(𝐪i + , … , 𝐪i(h))[𝑝 1 , … , 𝑝 𝑚 ] is any permutation of [1, … ,𝑚]
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 49
![Page 50: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/50.jpg)
¡ DeepSets architecture
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 50
𝐪+𝐪/…𝐪h
𝜙 mean 𝛽 𝐪
Permutation Invariant
Vector embeddings of the input queries
Features of the input queries
𝜙(𝐪+)
𝜙(𝐪h)
Vector embedding of the intersection query
![Page 51: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/51.jpg)
¡ Key Idea: embed queries in vector space“Where did Canadian citizens with Turing Award graduate?”
Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 51
Computation Graph Embedding Space
Win
HintonBengio
Pearl
TuringAward
CanadaCitizen
NYU
Graduate
McGill Edinburgh
Cambridge
𝐪
𝐪 = ℐ(𝐪+, 𝐪/)
TuringAward
Canada
Projection
ProjectionProjection
Intersection
Intersection
Trudeau Bieber
𝐪/
𝐪+
Answers!
![Page 52: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/52.jpg)
¡ Given an entity embedding 𝐯 and a query embedding 𝐪, the distance is 𝑓e 𝑣 = ||𝐪 − 𝐯||.
¡ Trainable parameters: § entity embeddings: 𝑑 𝑉§ relation embeddings: 𝑑 𝑅§ intersection operator 𝜙, 𝛽: number of parameters
does not depend on graph size
¡ Same training strategy as TransE
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 52
![Page 53: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/53.jpg)
¡ Training:1. Sample a query 𝑞, answer 𝑣, negative sample 𝑣′.2. Embed the query 𝐪.3. Calculate the distance 𝑓e(𝑣) and 𝑓e(𝑣F).4. Optimize the loss ℒ.
¡ Query evaluation:1. Given a test query 𝑞, embed the query 𝐪.2. For all 𝑣 in KG, calculate 𝑓e(𝑣).3. Sort the distance and rank all 𝑣.
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 53
![Page 54: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/54.jpg)
¡ Taking the intersection between two vectors is an operation that does not follow intuition.
¡ When we traverse the KG to achieve the answers, each step produces a set of reachable entities. How can we better model these sets?
¡ Can we define a more expressive geometry to embed the queries?
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 54
![Page 55: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/55.jpg)
1. Introduction to Knowledge Graphs
2. Knowledge Graph completion
3. Path Queries
4. Conjunctive Queries
5. Query2Box: Reasoning with Box Embeddings
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 55
![Page 56: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/56.jpg)
¡ Embed queries with hyper-rectangles (boxes)𝐪 = (𝐶𝑒𝑛𝑡𝑒𝑟 𝑞 , 𝑂𝑓𝑓𝑠𝑒𝑡(𝑞))
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 56
Edinburgh𝑞McGill
Cambridge
Stanford
Embedding Space
![Page 57: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/57.jpg)
¡ Taking intersection between two vectors is an operation that does not follow intuition.§ Intersection of boxes is well-defined!
¡ When we traverse the KG to achieve the answers, each step produces a set of reachable entities. How can we better model these sets? § Boxes are a powerful abstraction, as we can
project the center and control the offset to model the set of entities enclosed in the box.
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 57
![Page 58: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/58.jpg)
¡ Parameters: § entity embeddings: 𝑑 𝑉
§ entities are seen as zero-volume boxes
§ relation embeddings: 2𝑑 𝑅§ augment each relation with an offset
§ intersection operator 𝜙, 𝛽: number of parameters does not depend on graph size § New operator, inputs are boxes and output is a box
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 58
![Page 59: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/59.jpg)
¡ Embed queries in vector space“Where did Canadian citizens with Turing Award graduate?”Note that computation graph stays the same!Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 59
Computation Graph
TuringAward
Canada Canada
TuringAward
Embedding Space
![Page 60: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/60.jpg)
¡ Embed queries in vector space“Where did Canadian citizens with Turing Award graduate?”Note that computation graph stays the same!Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 60
Computation Graph
TuringAward
Canada
Projection
ProjectionCanada
TuringAward
Embedding Process
?
![Page 61: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/61.jpg)
¡ Geometric Projection Operator 𝒫¡ 𝒫 : Box × Relation → Box
𝐶𝑒𝑛 𝑞F = 𝐶𝑒𝑛 𝑞 + 𝐶𝑒𝑛 𝑟𝑂𝑓𝑓 𝑞F = 𝑂𝑓𝑓 𝑞 + 𝑂𝑓𝑓(𝑟)
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 61
𝑞
𝑞′
𝑟
11/21/19
![Page 62: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/62.jpg)
¡ Embed queries in vector space“Where did Canadian citizens with Turing Award graduate?”Note that computation graph stays the same!Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 62
Computation Graph
TuringAward
Canada
Projection
Projection
Win
Citizen
Trudeau
HintonBengio
Pearl
Canada Bieber
TuringAward
Embedding Space
![Page 63: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/63.jpg)
¡ Embed queries in vector space“Where did Canadian citizens with Turing Award graduate?”Note that computation graph stays the same!Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 63
Computation Graph
TuringAward
Canada
Projection
Projection
Intersection
Intersection
Embedding Space
?
Win
Citizen
Trudeau
HintonBengio
Pearl
Canada Bieber
TuringAward
![Page 64: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/64.jpg)
¡ Geometric Intersection Operator ℐ¡ ℐ : Box ×⋯× Box → Box § The new center is a weighted average.§ The new offset shrinks.
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 6411/21/19
![Page 65: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/65.jpg)
¡ Geometric Intersection Operator ℐ¡ ℐ : Box ×⋯× Box → Box
𝐶𝑒𝑛 𝑞{3A|; =?{
𝒘{ ⊙ 𝐶𝑒𝑛 𝑞{
𝑂𝑓𝑓 𝑞{3A|;= min 𝑂𝑓𝑓 𝑞+ , … , 𝑂𝑓𝑓 𝑞3⊙ 𝜎(𝐷𝑒𝑒𝑝𝑠𝑒𝑡𝑠(𝐪+, … , 𝐪3))
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 65
weight
guarantees shrinking
Sigmoid function:squashes output in (0,1)
dimension-wise product
11/21/19
![Page 66: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/66.jpg)
¡ Embed queries in vector space“Where did Canadian citizens with Turing Award graduate?”Note that computation graph stays the same!Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 66
Computation Graph
TuringAward
Canada
Projection
Projection
Intersection
Intersection
Win
Citizen
Trudeau
HintonBengio
Pearl
Canada Bieber
TuringAward
Embedding Space
![Page 67: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/67.jpg)
¡ Embed queries in vector space“Where did Canadian citizens with Turing Award graduate?”Note that computation graph stays the same!Follow the computation graph:
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 67
Computation Graph
TuringAward
Canada
Projection
ProjectionProjection
Intersection
Intersection
Win
Citizen
Trudeau
HintonBengio
PearlGraduate
McGill
Edinburgh
Canada Bieber
CambridgeTuringAward
Embedding SpaceNYU
![Page 68: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/68.jpg)
¡ Given a query box 𝐪 and entity vector 𝐯,
𝑑��� 𝐪, 𝐯 = 𝑑��A 𝐪, 𝐯 + 𝛼 ⋅ 𝑑{3(𝐪, 𝐯)
where 0 < 𝛼 < 1.
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 68
𝑑��A(𝐪, 𝐯)𝑑{3(𝐪, 𝐯)
𝑣
𝐶𝑒𝑛(𝑞)
11/21/19
![Page 69: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/69.jpg)
¡ Given a set of queries and answers,ℒ = − log 𝜎 𝛾 − 𝑑��� 𝑞, 𝑣 −
log 𝜎(𝑑��� 𝑞, 𝑣{F − 𝛾)
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 69
𝛾 𝛾
− log 𝜎 𝛾 − 𝑑��� 𝑞, 𝑣minimize loss → minimize 𝑑���(𝑞, 𝑣)
− log 𝜎 𝑑��� 𝑞, 𝑣′ − 𝛾minimize loss → maximize 𝑑���(𝑞, 𝑣′)
![Page 70: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/70.jpg)
¡ Can query2box handle different relation patterns?
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 70
Embedding Symmetry Composition One-to-many
TransE û ü û
TransH ü û ü
Query2Box ü ü ü
For details please check the paper https://openreview.net/forum?id=BJgr4kSFDS
![Page 71: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/71.jpg)
¡ 1-to-N, N-to-1, N-to-N relations.¡ Example: Both (ℎ, 𝑟, 𝑡+) and (ℎ, 𝑟, 𝑡/) exist.
¡ Box Embedding can handle since 𝑡+ and 𝑡/ will be mapped to different locations in the box of (ℎ, 𝑟). ü
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 71
𝐡
𝐭+𝐭/
𝐡 + 𝐫
![Page 72: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/72.jpg)
¡ Symmetric Relations:𝑟 ℎ, 𝑡 ⇒ 𝑟 𝑡, ℎ ∀ℎ, 𝑡
¡ Example: Family, Roommate¡ Box Embedding
𝐶𝑒𝑛 𝑟 = 0ü
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 72
𝐡𝐭
𝐡 + 𝐫
For symmetric relations 𝑟, we could assign 𝐶𝑒𝑛 𝑟 = 0. In this case, as long as 𝑡 is in the box of (ℎ, 𝑟), it is guaranteed that ℎ is in the box of (𝑡, 𝑟). So we have 𝑟(ℎ, 𝑡) ⇒ 𝑟(𝑡, ℎ)
𝐭 + 𝐫
![Page 73: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/73.jpg)
¡ Composition Relations:𝑟+ 𝑥, 𝑦 ∧ 𝑟/ 𝑦, 𝑧 ⇒ 𝑟1 𝑥, 𝑧 ∀𝑥, 𝑦, 𝑧
¡ Example: My mother’s husband is my father.¡ Box Embedding
𝐫1 = 𝐫+ + 𝐫/ ü
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 73
𝐱𝐫1
𝐲
𝐳
𝐱 + 𝐫𝟏𝐲 + 𝐫𝟐
𝐱 + 𝐫𝟏 + 𝐫𝟐
For composition relations, if 𝑦 is in the box of (𝑥, 𝑟+) and 𝑧 is in the box of (𝑦, 𝑟/), it is guaranteed that 𝑧 is in the box of (𝑥, 𝑟+ + 𝑟/).
![Page 74: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/74.jpg)
¡ Can we embed even more complex queries?“Where did Canadians with Turing Award or Nobel graduate?”
¡ Conjunctive queries + disjunction is called Existential Positive First-order (EPFO) queries.
¡ Can we also design a disjunction operator and embed EPFO queries in low-dimensional vector space? YES!
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 74
For details please check the paper https://openreview.net/forum?id=BJgr4kSFDS
![Page 75: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/75.jpg)
¡ Datasets: FB15K, FB15K-237
¡ Goal: can the model discover true answers that cannot be achieved by traversing the KG?§ Training KG: Training Edges§ Validation KG: Training Edges + Validation Edges§ Test KG: Training Edges + Validation Edges + Test Edges
¡ Queries:
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 893i2i3p2p1p
Training Conjunctive Queries
ip pi
Unseen Conjunctive Queries
uu
uu
2u up
Union Queries
11/21/19
![Page 76: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/76.jpg)
¡ Given a query structure, use pre-order traversal (traverse from root to leaves) to assign an entity/relation for every node/edge.
¡ We explicitly rule out degenerated queries.
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 90
rootleaf
leaf
𝒓 𝒓�𝟏
𝑉 𝑉?𝑣\
𝒓
𝒓𝑣\
𝑣\
𝑉?
11/21/19
![Page 77: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/77.jpg)
¡ After instantiation, run post-order traversal (traverse from leaves 𝑣+, 𝑣/ to root) to achieve all answers.
¡ For test queries, we guarantee that they cannot be fully answered on training/validation KG.
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 91
𝑣+
𝑣/
𝑟+
𝑟/
𝑟1
root
11/21/19
![Page 78: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/78.jpg)
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 92
3i2i3p2p1p
Training Conjunctive Queries
ip pi
Unseen Conjunctive Queries
uu
uu
2u up
Union Queries
11/21/19
![Page 79: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/79.jpg)
¡ What does query2box actually learn?
Example: “List male instrumentalists who play string instruments”
¡ We use T-SNE to reduce the embedding space to a 2-dimensional space, in order to visualize the query results
11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 93
![Page 80: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/80.jpg)
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 94
“List male instrumentalists who play string instruments”StringInstrument
Male
Projection
Projection
Projection Intersection
Intersection
Embedding of 14951 entities
11/21/19
![Page 81: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/81.jpg)
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 95
Anchor
“List male instrumentalists who play string instruments”StringInstrument
11/21/19
![Page 82: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/82.jpg)
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 96
TP
FPTN
FN
“List male instrumentalists who play string instruments”StringInstrument Projection
TPR: 100%FPR: 0%
# of string instruments: 10
11/21/19
![Page 83: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/83.jpg)
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 97
TP
FPTN
FN
“List male instrumentalists who play string instruments”StringInstrument Projection Projection
# of instrumentalists: 472
TPR: 98.4%FPR: 0.01%
11/21/19
![Page 84: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/84.jpg)
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 98
“List male instrumentalists who play string instruments”
Male
Anchor
11/21/19
![Page 85: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/85.jpg)
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 99
TP
FPTN
FN
MaleProjection
TPR: 97.9%FPR: 0.01%
“List male instrumentalists who play string instruments”
# of men: 3555
11/21/19
![Page 86: CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu ...web.stanford.edu/class/cs224w/slides/17-knowledge.pdf · ¡Edges in KG are represented as triples(ℎ,$,%) §head(ℎ)has](https://reader034.vdocuments.mx/reader034/viewer/2022042307/5ed35cad940a9644e57e1608/html5/thumbnails/86.jpg)
Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 100
TP
FPTN
FN
StringInstrument
Male
Projection
Projection
Projection Intersection
Intersection
“List male instrumentalists who play string instruments”
# of answers: 396
TPR: 99.4%FPR: 0.01%
11/21/19