Upload File

recurrent convolutional neural networks for text classification

  • Home
  • Technology
  • Recurrent Convolutional Neural Networks for Text Classification
15
Recurrent Convolutional Neural Networks for Text Classification Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao AAAI 2015 読み手：周 双双 6/30/16 1

Upload: shuangshuang-zhou

Post on 14-Apr-2017

759 views

Category:

Technology


5 download

Report

TRANSCRIPT

Page 1: Recurrent Convolutional Neural Networks for Text Classification

RecurrentConvolutionalNeuralNetworksforTextClassification

Siwei Lai,Liheng Xu,KangLiu,JunZhaoAAAI2015

読み手：周 双双

6/30/16 1

Page 2: Recurrent Convolutional Neural Networks for Text Classification

Background- Textclassification◦ Featurerepresentationinpreviousstudies:bag-of-words(BoW)model,whereunigrams,bigrams,n-gramsorsomeexquisitelydesignedpatterns.

◦ Disadvantages:“ignorethecontextualinformationorwordorderintextsandremainunsatisfactoryforcapturingthesemanticsofthewords.”

Wordembeddinganddeepneuralnetwork

6/30/16 2

Page 3: Recurrent Convolutional Neural Networks for Text Classification

WhyRecurrentConvolutionalNeuralNetwork?◦ RecursiveNeuralNetwork(RecursiveNN)◦ Disadvantage:”ItsperformanceheavilydependsontheperformanceofthetextualtreeexhibitsatimecomplexityofatleastO(n2),wherenisthelengthof thetext.”

◦ RecurrentNeuralNetwork(RecurrentNN)◦ Disadvantage: “abiasedmodel,wherelaterwordsaremoredominant thanearlierwords”

◦ ConvolutionalNeuralNetwork(CNN)◦ Advantages:◦ capturethesemanticoftextsbetterthanrecursiveorrecurrentNN.◦ TimecomplexityofCNNisO(n).

◦ Disadvantages:◦ Itisdifficult todetermine thewindowsize

RecurrentConvolutional NeuralNetwork(RCNN):Learnmorecontextualinformation thanconventionalwindow-basedneuralnetworks

6/30/16 3

Page 4: Recurrent Convolutional Neural Networks for Text Classification

RecurrentConvolutionalNeuralNetworkWordRepresentationLearning

TextRepresentationLearning

6/30/16 4

Page 5: Recurrent Convolutional Neural Networks for Text Classification

WordRepresentationLearningTherepresentationofwordwi istheconcatenationoftheleft-sidecontextvectorcl(wi ),thewordembeddinge(wi )andtheright-sidecontextvectorcr(wi )

• Theleft-sidecontextvectorcl(wi ) andtheright-sidecontextvectorcr (wi )arecalculatedbythesimilarway.

• e(wi -1)isthewordembeddingofwordwi-1 ,cl(wi -1)istheleft-sidecontextvectorofthepreviouswordwi -1

• e(wi+1)isthewordembeddingofwordwi+1 ,cr (wi +1)istheright-sidecontextvectorofthenextwordwi +1

6/30/16 5

Page 6: Recurrent Convolutional Neural Networks for Text Classification

RecurrentConvolutionalNeuralNetworkWordRepresentationLearning

TextRepresentationLearning

• Applyalineartransformationtogetherwiththetanh activationfunctiontoxi andsenttheresulttothenextlayer.

• yi(2)isalatentsemanticvector,willbeusedtodeterminethemostusefulfactorforrepresentingthetext.

6/30/16 6

Page 7: Recurrent Convolutional Neural Networks for Text Classification

TextRepresentationLearningWordRepresentationLearning

TextRepresentationLearning

• Amax-pooling layerisappliedonalltherepresentationsofwords.

• Avariouslengthstextisconvertedintoafixed-lengthvector.

• Thek-th elementofy(3)isthemaximuminthek–th elementsofyi(2)

Whymaxpooling, notaveragepooling?• Aimtofind themostimportantlatent

semanticfactorsinthetexts.

6/30/16 7

Page 8: Recurrent Convolutional Neural Networks for Text Classification

OutputlayerWordRepresentationLearning

TextRepresentationLearning

Softmax function isappliedtoy(4)

6/30/16 8

Page 9: Recurrent Convolutional Neural Networks for Text Classification

Trainingnetworkparameters

Theydefinealloftheparameterstobetrainedas θ

Thetraining targetofthenetworkisusedtomaximizethelog-likelihood withrespectto θ，whereDisthetrainingdocumentsetandclassD isthecorrectclassofdocumentD.

Stochasticgradientdescentisusedtooptimizethetrainingtarget.

6/30/16 9

Page 10: Recurrent Convolutional Neural Networks for Text Classification

DatasetandExperimentssettings

MeasureMarco-F1AccuracyAccuracyAccuracy

• Choseacommonusedhyper-parametersfollowing previousstudies(Collobert etal.2011;Turian,Ratinov,andBengio 2010)

• Setlearning rateα as0.01.Hidden layersizeHas100,wordembedding dimension as50,contextvectordimension as50.

• Wordembedding arerespectivelypre-trainedonEnglishandChineseWikipediadumpsbythedefaultparameterinword2vecwiththeskip-gramalgorithm.

6/30/16 10

Page 11: Recurrent Convolutional Neural Networks for Text Classification

ExperimentsResults– ComparisonofMethods

• Bag-of-words(BoW)/Bigram +logisticregression(LR)/SVM (WangandManning2012

• AverageEmbedding+LR: atf-idf weightedaverageofthewordembeddings andsubsequently appliesasoftmax layer

• LDA:ClassifyLDA-EMandLabeled-LDA• TreeKernels:PostandBergsma(2013)used

varioustreekernelsasfeatures.Here,theycomparedcontext-freegrammar(CFG)producedbyBerkeleyparserandtherankingfeaturesetofC&J.

• RecursiveNN:tworecursive-basedmethods.RecursiveNN (Socher etal.2011a)andRecursiveNeuralTensorNetworks(RNTNs)(Socher etal.2013)

• CNN:aconvolutionkernelconcatenatedthewordembeddings inapre-definedwindow inCollobert etal.2011.

• Paragraph-Vector:LeandMikolov 2014.6/30/16 11

Page 12: Recurrent Convolutional Neural Networks for Text Classification

ExperimentsResultsandDiscussion

6/30/16 12

• Neuralnetworkapproaches(RecursiveNN,CNN,RCNN)outperformthetraditionalmethods(e.g.BoW+LR)forallfourdatasets.Neuralnetworkscancapturemorecontextualinformation offeatures.

• RecursiveNN outperformed CNN,RCNNonSSTdataset.Theimportanceofmax-pooling layerandconvolutional layer.RNTN(3-5hours)vs.RCNN(severalminutes)

• ComparingRCNNwithtreekernelsmethods,RCNNmightbeuseful inlow-resourcelanguages.

• RCNNvs.CNN.Recurrentstructureisbetterthanwindow-basedstructure.

Page 13: Recurrent Convolutional Neural Networks for Text Classification

RCNNvs.CNN

6/30/16 13

• Theyconsideralloddwindowsizefrom1to19totrainandtesttheCNNmodel.

• RCNNoutperformstheCNNforallwindowsizesanditdoesnotrelyonthewindowsize.

• Therecurrentstructurecanpreservelongercontextualinformationandintroduceslessnoise.

Page 14: Recurrent Convolutional Neural Networks for Text Classification

LearnedKeywords

6/30/16 14

• Themax-pooling layerselectsthemostimportantwordsintexts.

• Presentthecenterwordanditsneighboring trigram.

• Listthecomparison resultswithRNTN(Socher etal.2013)onsentimentalclassification.

Page 15: Recurrent Convolutional Neural Networks for Text Classification

感想

◦ RCNNDoesrelyonasyntacticparser.◦ RCNNmightbeusefulinlow-resourcelanguages.◦ RCNNhasalowertimecomplexityofO(n).◦ LimitationofRCNN?◦ Theirmodelcancapturethekeycomponentsintextsfortextclassification

Capturethekeycomponentsfordisambiguation?PlantoapplyRCNNonothertasks

6/30/16 15

Sourcecodes:https://github.com/knok/rcnn-text-classification


Learning representations from EEG with Deep Recurrent Convolutional neural networks

Convolutional Recurrent Neural Networks for Polyphonic Sound … · 1 Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection Emre C¸akır, Giambattista Parascandolo,

Convolutional Recurrent Neural Networks for Dynamic MR ... · 1 Convolutional Recurrent Neural Networks for Dynamic MR Image Reconstruction Chen Qin*y, Jo Schlemper*, Jose Caballero,

Lecture 10: Neural Networks and Deep Learningsaravanan-thirumuruganathan.github.io/cse5334... · Neural Networks Deep Learning Convolutional Neural Networks Recurrent Neural Networks

Stock Market Trend Prediction Using Recurrent Convolutional Neural …tcci.ccf.org.cn/conference/2018/papers/145.pdf · 2018-08-16 · Stock Market Trend Prediction Using Recurrent

Diffusion Convolutional Recurrent Neural Network: Data

Convolutional Neural Networks (CNNs) and Recurrent Neural

Convolutional Neural Networks Analyzed via Convolutional ... · PDF fileConvolutional Neural Networks Analyzed via Convolutional Sparse Coding ... Deep Learning, Convolutional Neural

Cumulative Damage Modeling with Recurrent Neural Networks · machines; convolutional neural networks; and purely data-driven versions of recurrent neural networks, including the long

Super-resolution recurrent convolutional neural networks

Convolutional Neural Networks · 2017-10-10 · –Lai S, Xu L, Liu K, et al. Recurrent Convolutional Neural Networks for Text Classification[C]//AAAI. 2015, 333: 2267-2273. –Pinheiro

Look Closer to See Better: Recurrent Attention Convolutional … · 2017. 5. 31. · Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-grained Image

Sequence Modelling and Recurrent Neural Networks (RNNs) · •Neural Networks (recap) and Deep Learning •Improving DNN: Hyperparameter tuning, regularization, optimization •Convolutional

SqueezeSeg: Convolutional Neural Nets with Recurrent … · SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud

Deep convolutional tensor network€¦ · A layer of a convolutional neural network ... defined by a convolution kernel, to all sliding windows of an input. A recurrent neural net-work

Convolutional Neural Networks - unipi.it · 2020-02-14 · •Convolutional Neural Networks •Deep Autoencoders and RBM •Gated Recurrent Networks (LSTM, GRU, …) •Advanced topics

Sketch Classification with Neural Networks1218490/FULLTEXT01.pdf · 2018-06-13 · The two types of neural networks studied are convolutional neural networks (CNN) and recurrent neural

Recurrent Fusion Network for Image Captioning arXiv:1807 ... · encoder is usually a convolutional neural network (CNN), while the decoder is a recurrent neural network (RNN). The

Convolutional-Recurrent Neural Network による自己運動識別mprg.jp/data/MPRG/F_group/F172_kamiya2016.pdf · Convolutional-Recurrent Neural Network による自己運動識別

Recurrent Residual Convolutional Neural Network based on U-Net … · 2018-02-21 · used for semantic image segmentation tasks. For example, a fully-connected convolutional neural

Convolutional Neural Networks Analyzed via Convolutional ...yromano/posters/ml_csc_poster.pdf · Convolutional Neural Networks ... Recently, convolutional sparse coding (CSC)

Recurrent Convolutional Neural Networks for Scene …proceedings.mlr.press/v32/pinheiro14.pdf · Recurrent Convolutional Neural Networks for Scene Labeling 4 4 2 2 2 2 Figure 1. A

Neural Networks - Seoul National Universitymrl.snu.ac.kr/courses/CourseMLforCG/NeuralNetworks.pdf · 2018-03-26 · •Neural network basics •Convolutional Neural Networks •Recurrent

Chapter 7 Towards Automatically-Tuned Deep Neural Networksnetworks, such as convolutional or recurrent neural networks, to future work. To have access to neural network techniques

Recurrent Convolutional Neural Network MSER-Based Approach

CONVOLUTIONAL-RECURRENT NEURAL NETWORKS ...hzhao1/papers/ICASSP2018/...tional recurrent neural networks (BRNN) that have recurrent connections in both directions. The output of the

Deep Learning for Neural Networks - viXra · Deep Learning for Neural Networks Today, deep neural networks with different architectures, such as convolutional, recurrent and autoencoder

Deep Recurrent and Convolutional Neural Networks for ... · PDF fileDeep Recurrent and Convolutional Neural Networks for Automated Behavior Classification ... This thesis focuses

Urs Köster - Convolutional and Recurrent Neural Networks

Video Compression Using Recurrent Convolutional …cs231n.stanford.edu/reports/2017/pdfs/423.pdfVideo Compression Using Recurrent Convolutional Neural Networks Cedric Yue Sik Kin Electrical

Recurrent Convolutional Neural Networks for …openaccess.thecvf.com/content_cvpr_2017/papers/Cui...Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition

Speech Emotion Recognition Using Deep Convolutional ...Index Terms—Convolutional Neural Network, Speech Emotion Recognition, Simple Recurrent Unit I. INTRODUCTION anguage, as one

IMPLEMENTASI ALGORITMA CONVOLUTIONAL RECURRENT …

Accelerating Convolutional Neural Network Systemshgrg1/publications/honours.pdf · 2015-07-29 · Accelerating Convolutional Neural Network Systems ... Convolutional Neural Networks

Diffusion Convolutional Recurrent Neural Network Data ...yaguang/papers/dcrnn_slides.pdfYaguang Li (USC)Author Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic