reporter: weibo gaohome.ustc.edu.cn/~huangzhy/files/slides/zhenyahuang-sigir2020s-… · reporter:...
TRANSCRIPT
![Page 1: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/1.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 1
Reporter: Weibo Gao
![Page 2: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/2.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 2
Outline
Background1
2 Problem Definition
Framework3
Experiment4
Conclusion & Future work5
![Page 3: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/3.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 3
BackgroundØAutomatically answering math problems
Ø A crucial and challenging task in AIØ Requirements
Ø Linguistic understanding abilityØ Semantic understandingØ Operator extraction
Ø Mathematical comprehension abilityØ Understand formulas with free-text format
![Page 4: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/4.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 4
Related workØMath word problem
Ø Elementary problem (primary school level)Ø Translate questions text into expression forms for answersØ Existing methods
Ø Rules-schemes-matching methodsØ Statistical learning
Ø E.g., template-based, tree-basedØ Seq2seq deep learning
Gwen was organizing her book case making sure each of the shelves had exactly 9 books on it. She has 2 types of books - mystery books and picture books. If she had 3 shelves of mystery books and 5 shelves of picture books.How many books did she have total?
Math word problem expression
(3 5) 9 72+ ´ =
expression tree
Just consist of natural language content
![Page 5: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/5.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 5
BackgroundØMath word Problem
Ø Elementary problem (primary school level)Ø Linguistic learning for natural language contentØ Operator extraction (+)Ø Semantic understanding
ØMathematical problemØ Complex problem (high school level)
Ø Language contentØ Specific but informative formulas
Ø RequirementØ Linguistic understandingØ Mathematical comprehension
Math word problem Mathematical problem
![Page 6: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/6.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 6
BackgroundØChallenges: How to represent formula-enriched problem?
Ø How to to understand formulas with their free-text format?
Ø How to design a unified architecture to incorporate linguistic and mathematical information?
!sin %2
sin, √, x, /, 2
s, i, n, √, x, /, 2
word-level
character-level
Mathematical information
Linguistic information
\sin, \sqrt, \frac
![Page 7: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/7.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 7
Outline
Background1
2 Problem Definition
Framework3
Experiment4
Conclusion & Future work5
![Page 8: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/8.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 8
Problem DefinitionØGiven
Ø Mathematical problem:Ø Token: is a word token or formula token (e.g., quantities, symbols)
ØGoalØ Read tokens from Ø Gnerate answer sequence:
{ }1 2, , , LP p p p= !ip
P{ }1 2, , , TY y y y= !
Answer: 30
3 0= { }Y1Y 2Y
Answer sequence YMathematical problem P
Problem: Let 3 + x = 13 . Solve x .
Let 3 + x ... Solve x .Let 3 + x= { }P1wP 2
fP 3fP 4
fP 9wP 10
fP 11wP...
Fomulas
: word token : fomula tokenwiP
fiP
![Page 9: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/9.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 9
Outline
Background1
3 Framework
Problem Definition2
Experiment4
Conclusion & Future work5
![Page 10: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/10.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 10
NMS FrameworkØNMS framework
Ø Formula Graph ConstructionØ Develop an assistant tool to construct formula dependency graph
Ø Neural SolverØ FGN: Formula graph networkØ Sequence model: Encoder-Decoder architecture
Semantic space
Mathematical space
![Page 11: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/11.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 11
NMS FrameworkØFormula graph construction
Ø Goal: present formulas in a structural wayØ Develop a TeX-based formula-dependent graph toolØ Nodes
Ø Variables: Ø Numbers: 2Ø Operators: \tan
Ø Edges (four relasions)Ø Brother, father, childØ Relative
Ø FeaturesØ Attribute, content
Ø Reduce redundantØ Keep structure informationØ Enhance semantic information
Advantages
q
![Page 12: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/12.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 12
NMS FrameworkØNeural solver
Ø FGN: capture fomula structure informationØ Sequence model: incorporate semantic and structural information
Neural solver
Formula Graph Network
![Page 13: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/13.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 13
Outline
Background1
4 Experiment
Problem Definition2
Framework3
Conclusion & Future work5
![Page 14: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/14.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 14
ExperimentØDataset
Ø MATH dataset (high school level)ØData analysis
Ø Formula tokens take large portionsØ 69% on averageØ Larger portions in shorter problems
ØBaseline methods (seq2seq)Ø GRUØ BiGRUØ RMCØ AttentionØ Transformer
ØEvaluation metricsØ ACC, BLEU, ROUGE
![Page 15: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/15.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 15
ExperimentØExperiment
Ø Task: solving mathematical problemsØ Observations
Ø NMS performs the bestØ Capture mathematical relations effectively
Ø Transformer and Seq2Seq-BiGRU perform better than other baselinesØ Design sophisticated encoders
Ø RMC performs not very wellØ Probably because it requires many parameters
![Page 16: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/16.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 16
ExperimentØVisualization
Ø Task: project problems embeddings into 2D space by t-SNEØ Observations
Ø Problems with same concepts learned are easier to be groupedØ They are closer in the hidden space
Ø Problems with simple formula structures cluster nearlyØ E.g., “Set” problems
Ø Many types of formulas cause different patternsØ E.g., “Function” problems
More reasonable
![Page 17: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/17.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 17
Outline
Background1
5 Conclusion & Future work
Problem Definition2
Framework3
Experiment4
![Page 18: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/18.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 18
Conclusion & Future workØOverall results
Ø Develop a TeX-based formula-dependent graph tool to maintain the structural information of each problem.
Ø Design FGN to capture mathematical relations.Ø Design a neural solver to incorporate semantic infomation and structural
infomation.
ØFuture workØ Seek ways to predict quantities effectively
Ø ½ vs. ⁄""" ###Ø Design different graph networks for learning formula structure
Ø Reasoning on different problem typesØ Consider more specific structures of more complex problems
Ø “geometry” problem: containing figures
![Page 19: Reporter: Weibo Gaohome.ustc.edu.cn/~huangzhy/files/slides/ZhenyaHuang-SIGIR2020s-… · Reporter: Weibo Gao. Anhui Province Key Laboratory of Big Data Analysis and Application 2](https://reader035.vdocuments.mx/reader035/viewer/2022062509/6043641dee4b36549f0dc58f/html5/thumbnails/19.jpg)
Anhui Province Key Laboratory of Big Data Analysis and Application 19
Thanks for your listening!