Slide 1

Creating Consistent Scene Graphs Using a Probabilistic Grammar Tianqiang LiuSiddhartha ChaudhuriVladimir G. Kim Qi-Xing HuangNiloy J. MitraThomas Funkhouser 11,23 3,451 123 4 5 Princeton University Cornell University Stanford University TTICUCL Slide 2 Motivation Growing number of 3D scenes online. Google 3D warehouse Slide 3 Motivation Synthesis [Fisher et al 2012, Xu et al 2013] Understanding [Xu et al 2014, Song et al 2014] Slide 4 Goal Input: A scene from Trimble 3D Warehouse Slide 5 Goal Output 1: Semantic segmentations Slide 6 Goal Output 2: Category labels. Door Nightstand Window Mattress Bed frame Pillow Dresser Mirror Heater Artifact Slide 7 Goal Output 2: Category labels at different levels. Bed & supported Dresser & supported Slide 8 Goal Sleeping area Vanity area Door set Output 2: Category labels at different levels. Slide 9 Challenges Shape is not distinctive. Night table Console table Slide 10 Challenges Contextual information Meeting chair Meeting table Study desk Study chair Slide 11 Challenges All-pair contextual information is not distinctive. #pairs feet #pairs feet Slide 12 Challenges Slide 13 Slide 14 Slide 15 All-pair contextual information is not distinctive. #pairs feet #pairs feet Slide 16 Key Idea Semantic groups Semantic hierarchy Slide 17 Key Idea #pairs feet #pairs feet Slide 18 Pipeline Probabilistic grammar Slide 19 Related Work Van Kaick et al. 2013 Slide 20 Related Work Van Kaick et al. 2013Boulch et al. 2013 Slide 21 Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results Slide 22 Probabilistic Grammar Labels Rules Probabilities Slide 23 Labels bed, night table, sleeping area Examples: Slide 24 Rules sleeping areabed, night table Example: Slide 25 Probabilities Derivation probabilities Cardinality probabilities Geometry probabilities Spatial probabilities Slide 26 Derivation probability bedbed frame, mattress Slide 27 Cardinality probability sleeping areabed, night table Slide 28 Geometry probability Slide 29 Spatial probability Slide 30 Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results Slide 31 Pipeline Probabilistic grammar Slide 32 Learning a Probabilistic Grammar Identify objects Speed X 5 Slide 33 Learning a Probabilistic Grammar Label objects Speed X 5 Slide 34 Learning a Probabilistic Grammar Group objects Speed X 5 Slide 35 Learning a Probabilistic Grammar Grammar generation Labels all unique labels Rules Probabilities Slide 36 Learning a Probabilistic Grammar Grammar generation Labels Rules concatenating all children for each label Probabilities Nightstand & supported NightstandTable lamp Nightstand & supported NightstandPlant Nightstand & supported NightstandTable lamp Plant Training example 1:Training example 2: Slide 37 Learning a Probabilistic Grammar Grammar generation Labels Rules Probabilities : learning from occurrence statistics : estimating Gaussian kernels : kernel density estimation Slide 38 Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results Slide 39 Pipeline Probabilistic grammar Slide 40 Pipeline Probabilistic grammar Slide 41 Scene parsing Objective function is the unknown hierarchy is the input scene is the probabilistic grammar Slide 42 Scene parsing After applying Bayes rule Slide 43 Scene parsing After applying Bayes rule Prior of hierarchy Slide 44 Scene parsing After applying Bayes rule Prior of hierarchy : probability of a single derivation formulated using Slide 45 Scene parsing After applying Bayes rule Prior of hierarchy compensates for decreasing probability as has more internal nodes. Slide 46 Scene parsing After applying Bayes rule Likelihood of scene Slide 47 Scene parsing After applying Bayes rule Likelihood of scene : geometry probability Slide 48 Scene parsing After applying Bayes rule Likelihood of scene : sum of all pairwise spatial probabilities Slide 49 Scene parsing We work in the negative logarithm space Slide 50 Scene parsing Rewrite the objective function recursively where is the root of, is the energy of a sub-tree. XX Slide 51 Scene parsing The search space is prohibitively large Problem 1: #possible groups is exponential. Problem 2: #label assignments is exponential. Slide 52 Scene parsing Problem 1: #possible groups is exponential. Slide 53 Solution: proposing candidate groups. Scene parsing Slide 54 Rule: a a c c d d b b a a d d b b c c b b d d a a c c c c a a d d b b Problem 2: #label assignments is exponential. Slide 55 Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization where is partial label of Slide 56 Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization where Now #rules and #assignments are both polynomial. The problem can be solved by dynamic programming. is partial label of Slide 57 Scene parsing Problem 2: #label assignments is exponential. Solution: bounding #RHS by grammar binarization Convert the result to a parse tree of the original grammar Slide 58 Overview Grammar Structure Learning a Probabilistic Grammar Scene Parsing Results Slide 59 Benefit of hierarchy Meeting table Shape only Study chair Slide 60 Benefit of hierarchy Shape only Flat grammar Meeting chair Meeting table Study chair Slide 61 Benefit of hierarchy Shape only Flat grammar Study chair Study desk Meeting chair Meeting table Ours Meeting table Study chair Slide 62 Benefit of hierarchy Shape only Flat grammar Meeting chair Meeting table Ours Study area Meeting area Meeting table Study chair Slide 63 Benefit of hierarchy Shape only Chair Mattress Console table Slide 64 Benefit of hierarchy Shape onlyFlat grammar Chair Nightstand Console table Mattress Console table Slide 65 Benefit of hierarchy Shape onlyFlat grammarOurs Chair Nightstand Console table Mattress Console table Chair Desk Slide 66 Benefit of hierarchy Shape onlyFlat grammarOurs Chair Nightstand Console table Mattress Console table Study area Sleep area Slide 67 Datasets 77 bedrooms30 classrooms8 libraries 17 small bedrooms8 small libraries Slide 68 Benefit of hierarchy Object classification Slide 69 Generalization of our method Parsing Sketch2Scene data set Slide 70 Take-away message Modeling hierarchy improves scene understanding. Slide 71 Limitation and Future Work Modeling correlation in probabilistic grammar. Slide 72 Limitation and Future Work Modeling correlation in probabilistic grammar. Grammar learning from noisy data. Sleep area bed Nightstand & supported Nightstand Table lamp Input scene graphsGrammar Slide 73 Limitation and Future Work Applications in other fields. Modeling correlation in probabilistic grammar. Grammar learning from noisy data. Modeling from RGB-D data [Chen et al. 2014] Slide 74 Acknowledgement Data Kun Xu Discussion Christiane Fellbaum, Stephen DiVerdi Funding NSF, ERC Starting Grant, Intel, Google, Adobe Slide 75 Code and Data http://www.cs.princeton.edu/~tianqian/projects/hierarchy/ Slide 76 Back up slides Slide 77 Learning a Probabilistic Grammar Creating a training set manually 1.Identify leaf-level objects. 2.Provide a label for each object in a scene. 3.Group objects and provide group labels. 4.Maintain consistency. Sleeping area Nightstand & supported NightstandTable lamp Sleeping area Nightstand & supported Nightstand Slide 78 Sensitivity analysis Impact of training set size Impact of energy terms Slide 79 Benefit of hierarchy Object classification Object grouping